Page 1 of 1

Data over 20GB

Posted: Tue Feb 10, 2015 7:28 pm
by Ian Usher7109531

Good Evening,

I have a general question. I have a CSV file that i want to import into a database collection. I have stripped the majority of fields out of the CSV and the file size is still greater than 20mb.

I was just wondering if there was any way that we can get round this or is it a case of thats the maximum and thats it?

Any advice would be greatly appreciated.

Ian


Data over 20GB

Posted: Tue Feb 10, 2015 8:38 pm
by Evgene Karachevtsev

Hello Ian,

Depending on your current plan, you may have different amount of storage.
Could you please clarify what was your original file size ?


Data over 20GB

Posted: Tue Feb 10, 2015 8:41 pm
by Ian Usher7109531

Hi Evgene,

My original file size is 40mb. There are abiut 490000 rows in the csv file i want to create as a database collection.

Ian


Data over 20GB

Posted: Wed Feb 11, 2015 4:56 pm
by Evgene Karachevtsev

Ian,

You may thy this approach as a workaround:

  1. "cut" file by number of lines up to 20MB, "put" the remainder to the same file

  2. create a collection in a database for each file

  3. write sc-script, which will copy data from one collection to another, by 100 or 1000 lines

    We also consider the option of raising up the limit.


Data over 20GB

Posted: Wed Mar 04, 2015 8:26 am
by Ian Usher7109531

Hi Evgene,

in regards writing a script which will copy over one collection to another, do you have any tutorials or guidance on how you should do this, for example where you should write the code?

Even if you could just bullet point the steps needed to combine the data that would be greatly appreciated.

Many thanks,

Ian


Data over 20GB

Posted: Thu Mar 05, 2015 6:28 am
by Egor Kotov6832188

Hello Ian.

1) You can create a server-code script , if you need to copy files from one collection to another
1.1 Open Appery.io and click 'go to platform'
1.2 Find 'Servercode'
1.3 Click 'Create script',

but in your case, unfortunately, you need to make all manually.