Ian Usher7109531
Posts: 0
Joined: Tue Sep 16, 2014 6:57 pm

Data over 20GB

Good Evening,

I have a general question. I have a CSV file that i want to import into a database collection. I have stripped the majority of fields out of the CSV and the file size is still greater than 20mb.

I was just wondering if there was any way that we can get round this or is it a case of thats the maximum and thats it?

Any advice would be greatly appreciated.

Ian

Evgene Karachevtsev
Posts: 12
Joined: Mon Apr 28, 2014 1:12 pm

Data over 20GB

Hello Ian,

Depending on your current plan, you may have different amount of storage.
Could you please clarify what was your original file size ?

Ian Usher7109531
Posts: 0
Joined: Tue Sep 16, 2014 6:57 pm

Data over 20GB

Hi Evgene,

My original file size is 40mb. There are abiut 490000 rows in the csv file i want to create as a database collection.

Ian

Evgene Karachevtsev
Posts: 12
Joined: Mon Apr 28, 2014 1:12 pm

Data over 20GB

Ian,

You may thy this approach as a workaround:

  1. "cut" file by number of lines up to 20MB, "put" the remainder to the same file

  2. create a collection in a database for each file

  3. write sc-script, which will copy data from one collection to another, by 100 or 1000 lines

    We also consider the option of raising up the limit.

Ian Usher7109531
Posts: 0
Joined: Tue Sep 16, 2014 6:57 pm

Data over 20GB

Hi Evgene,

in regards writing a script which will copy over one collection to another, do you have any tutorials or guidance on how you should do this, for example where you should write the code?

Even if you could just bullet point the steps needed to combine the data that would be greatly appreciated.

Many thanks,

Ian

Egor Kotov6832188
Posts: 0
Joined: Wed Nov 19, 2014 5:15 pm

Data over 20GB

Hello Ian.

1) You can create a server-code script , if you need to copy files from one collection to another
1.1 Open Appery.io and click 'go to platform'
1.2 Find 'Servercode'
1.3 Click 'Create script',

but in your case, unfortunately, you need to make all manually.

Return to “Issues”