vector of Class objects vs vector of pointers to class objects.
-
Thanks to the two users "SeppJ" and "Gugelmoser". With their help, I have written a complete program which can read a data file and check different errors. However, this program can't read large data file. When I read large data with like 100,00,000 lines, it tries nearly 4-5 min then it says something like bad alloc(). I do not know why? Could anyone please hint me why this program is inefficient? Any alternative program to read such data would be very appreciable.
The file looks as follows is as follows.
- having rows with usually a certain fixed number of columns,
- different comments marked with the start letter #,
- also contains some blank lines.
- Blank lines and comment line should be ignored while reading.
- The file may also contain by mistake wrong number of columns which should be corrected first as well.
- Each row gives a class object and it should be saved as an element
vector of class object. - the columns in each row may be either separated by both white space and by tabs mixed.
file content:
1 snp1 1.1 1 2 snp2 1.2 2 3 snp3 1.3 3 4 snp4 1.4 4 # 1 #2 #hier #3 #test # # 5 snp5 1.5 5 #test 5 snp5 1.5 5 2 test 6 snp6 1.5 6 3ss snp7 #test #1 4 snp4 1.4 4 5 snp5 1.5 5 # # 7 snp7 1.5 7To read and display . I have written the following code.
#include<iostream> #include<fstream> #include<string> #include<sstream> #include<vector> #include<algorithm> #include<cstdlib> using namespace std; struct CSNP{ friend istream& operator>>(istream & is, CSNP& pCSNP); friend ostream& operator<<(ostream &out, const CSNP &pCSNP); string nchr; string snpName; float cm_pos; long int bp; CSNP():nchr(""),snpName(""), cm_pos(0.0),bp(0){} vector<CSNP>lociInfo; static void readMapFile(const string& fileName, vector<CSNP>locInfo); static void error(const string & ); static bool checkSNPinfo(const string& line, const long int& countSNPs); static bool three_columns; //~CSNP(); }; bool three_columns=false; void CSNP::error(const string& msg){ cerr<<msg <<endl; exit(1); } int main (int argc, char** argv){ if(argv[1]==0) CSNP::error("No file exits. please give argv[1].") ; vector<CSNP> genInfo; CSNP SNP; ifstream file(argv[1]); while(file>>SNP) genInfo.push_back(SNP); cout <<endl; cout<<"Total number of SNPs read: "<< genInfo.size()<<endl; cout << "---------Displaying SNP Info:----------------#\n"; for (unsigned int i=0; i<genInfo.size(); ++i) cout << genInfo[i]<< '\n'; cout <<endl; return 0;} ostream& operator<<(ostream &out, const CSNP &pCSNP) { return out << "chr No:"<< pCSNP.nchr<< ", SNP ID: "<< pCSNP.snpName << ", cm_pos: " << pCSNP.cm_pos<< " and bp: "<< pCSNP.bp; } bool checkSNPinfo(const string& line, const long int& countSNPs){ // This function will return true if there are four columns and if not then false: bool tfValue=false; //char* sline= (char*)line.c_str(); string buffer; vector<string> snp; istringstream line_parser(line); if(line_parser.fail()){ line_parser.unget(); line_parser.clear(ios_base::failbit); string count; stringstream ss; ss<<countSNPs; count=ss.str(); string msg="There is problem in reading "+count +"th line (excluding comments and blank lines) of your map file. Please correct or remove this line.\n"; CSNP::error(msg); } if(line_parser.good()) { //while(getline(line_parser,buffer)){ while(line_parser>>buffer) snp.push_back(buffer); cout << "snp.size() : " <<snp.size()<<endl; // cout << snp[0]<< " " <<snp[2]<<endl; if((snp.size()!=3) && (snp.size()!=4)) { string count; stringstream ss ; ss<<countSNPs; count=ss.str(); string msg="It is expected to have either 3 or 4 columns in "\ + count+\ "th line (excluding comments and blank lines) of your map file, but this is not the case here. Please either correct or delete this line. \n"; CSNP::error(msg); } if(!three_columns && (snp.size()==3)) { three_columns=true; } else if(!three_columns && snp.size()==4) { tfValue=true; } snp.resize(0); } return tfValue; } istream& operator >>(istream& is, CSNP& pCSNP){ static vector<bool>bool_snpCol;// this will check if all rows of map file has sam number of columns. string line; char cline[256]; static long int countSNPs=1; is>>ws; while(getline(is, line, '\n') && (!line.size() or line[0]=='#')); if(is.good()){ bool tfValue=checkSNPinfo(line,countSNPs); if(bool_snpCol.empty()) { bool_snpCol.push_back(tfValue); } if(bool_snpCol.size()<2) { bool_snpCol.push_back(tfValue); if(bool_snpCol[0]!= bool_snpCol[1]) { string count; stringstream ss ; ss<<countSNPs; count=ss.str(); string msg=" The " + count+"th row (excluding comments and blank lines)"+ "of your map file does not contain the same number of columns as in the previous rows. Please delete or correct this row.\n"; CSNP::error(msg); } bool_snpCol.pop_back(); } stringstream line_parser(line); if(!tfValue) // four columns? If yes true; line_parser>>pCSNP.nchr>>pCSNP.snpName>>pCSNP.bp; else line_parser>>pCSNP.nchr>>pCSNP.snpName>>pCSNP.cm_pos>>pCSNP.bp; if(line_parser.fail()){ line_parser.clear(ios_base::failbit); string count; stringstream ss ; ss<<countSNPs; count=ss.str(); string msg="There is a problem in "\ + count+\ "th row of your map file. Please either correct it or delete it. "; CSNP::error(msg); } is.exceptions(is.exceptions()|ios_base::badbit); if(line_parser.fail()){ //line_parser.unget(); line_parser.clear(ios_base::failbit); //return is; // end of file } } if(is.eof()){ //is.unget(); is.clear(ios_base::failbit); return is; // end of file } if(is.bad()) CSNP::error(" istream is bad. FATAL ERROR!"); //cout << "countSNPs: "<< countSNPs <<endl; countSNPs++; return is;}
-
please clean up the code and especially remove all comments which were placed by the guys to hint you what to do. also please choose variable names whcih are at least comprehensible, i.e more than 4 characters long.
At the moment, this is just abuse of the help offered by this forum.
-
Without having read much of your code, if you try to read 100 million lines at a time, you run out of space, which is exactly what your error message says. If you want to handle very big files, you have to find a way to read them chunk by chunk. If you need arbitrary access to any line, think about reading the lines just in time you need them. The proxy pattern might help you.
-
otze schrieb:
please clean up the code and especially remove all comments which were placed by the guys to hint you what to do. also please choose variable names whcih are at least comprehensible, i.e more than 4 characters long.
At the moment, this is just abuse of the help offered by this forum.thanks for your info. I haven't just copied and pasted the codes. I have worked my self and most of the comments were from me so that I can remember to work further on it.
but according to your suggestions I have deleted. it.thank you.
-
Michael E. schrieb:
Without having read much of your code, if you try to read 100 million lines at a time, you run out of space, which is exactly what your error message says. If you want to handle very big files, you have to find a way to read them chunk by chunk. If you need arbitrary access to any line, think about reading the lines just in time you need them. The proxy pattern might help you.
That is exactly what I also think but till now I haven't got idea how to do that way. Would you please give me some Literature names or internet webs where I can read about it? By the way there were 1 million lines not 100.
sorry
thank you
euklid
-
How many memory is used by your program before the crash?
-
euklid schrieb:
thanks for your info. I haven't just copied and pasted the codes. I have worked my self and most of the comments were from me so that I can remember to work further on it.
but according to your suggestions I have deleted. it.thank you.the intendation is still terrible misleading and makes the code hard to understand. Braces seem to be placed totally random. It looks more like copy & paste than real programmers work. This chaos makes it really hard to understand.
do you understand it?