[gradebook2-dev] some grades are missing when importing grades from .CSV files into GB2

[Thomas Amsler]

Are you sure that the user used Excel to edit the CSV file. I just cannot
imagine that Excel would produce an inconsistent data format. This looks to
me as if the user has hand edited the CSV file. One hint is that the CSV
file has Microsoft control characters for new lines, like as if the user
used MS NotePad to edit it.
- Thomas


On Tue, Sep 14, 2010 at 1:21 PM, Kim Huang <kimhuang at oit.rutgers.edu> wrote:

> The problem is the user never have a chance to verify all the commas and
> fields if they open it straightly from Microsoft Excel.
>
> If there is a requirement about the same number of commas for every line
> , the code somewhere should have a validation check and provide the
> error message to the user about the problem in .csv file.
>
> This problem was initially discovered by one of RU GB2 users, he
> probably used the same way as I created this file, which is from
> exporting using export function in GB2 then open in the Excel then fill
> the grades then importing, when he uploaded the grade file into GB2.
> There is no way that he knows he has to keep the same number of commas
> in all the fields, the file looks perfectly normal in Excel and it is
> exported originally from GB2.
>
> -Kim
>
>
> > Well I don't think its wrong to require the same number of comma
> > separated fields on all lines. So if line 1 has 10 fields, line 50
> > should have the same.
> >
> > Mike
> >
> > On Tue, Sep 14, 2010 at 11:12 AM, Kim Huang <kimhuang at oit.rutgers.edu
> > <mailto:kimhuang at oit.rutgers.edu>> <kimhuang at oit.rutgers.edu%3E%3E>wrote:
> >
> > The problem is in
> > ImportExportUtility.java, function : readInGradeDataFromImportFile
> > these lines:
> >
> -----------------------------------------------------------------------------
>
> >
> > String[] strippedColumns = new String[columns.length - 1];
> > int n = 0;
> > for (int i=0;i<columns.length;i++) {
> > Integer columnNumber = Integer.valueOf(i);
> >
> > if (ieInfo.getIgnoreColumns().contains(columnNumber))
> > continue;
> >
> > if (ieInfo.getCourseGradeFieldIndex() == i)
> >
> > continue;
> >
> >
> > if (n < strippedColumns.length)
> > strippedColumns[n] = columns[i];
> > n++;
> > --------------------------------
> > for line "a6 at 1.com <mailto:a6 at 1.com> <a6 at 1.com%3E> <
> mailto:a6 at 1.com>,a6 at 1.com <a6 at 1.com%3E,a6 at 1.com>
> > <mailto:a6 at 1.com%3E,a6 at 1.com> <a6 at 1.com%3E,a6 at 1.com%3E>
> > <mailto:a6 at 1.com>,88,,,,,," <a6 at 1.com%3E,88,,,,,,> <
> mailto:a6 at 1.com%3E,88,,,,,,> <a6 at 1.com%3E,88,,,,,,%3E>, the
> > size of columns is 8, so the size of
> > strippedColumns is defined as 7
> > the strippedColumns[n] = columns[i]; will execute , because n is 2
> > for
> > the grade "88" column,
> >
> > but for line "a7 at 1.com <mailto:a7 at 1.com> <a7 at 1.com%3E>, a7 at 1.com
> > <mailto:a7 at 1.com>,88" <a7 at 1.com%3E,88>, the size of columns is 3, so the
>
> > size of strippedColumns is defined as 2,
> > since n ==2 when the grade should be assigned, but if (n <
> > strippedColumns.length) returns false,
> > so the grade is not assigned.
> >
> > file Gradebook.csv looks perfectly normal when opened under Microsoft
> > excel , so I think a lot of faculties will have this type of the file
> > I don't know how important the size of strippedColumns in your code,
> > otherwise I will define it as the same size of columns
> >
> > Thanks,
> >
> > -Kim
> >
> >
> > Michael Wenk wrote:
> > > I think the last 3 lines aren't being imported because they
> > don't have
> > > all the columns the other lines have.
> > >
> > > Here are the last 4 lines:
> > >
> > > a6 at 1.com <mailto:a6 at 1.com> <a6 at 1.com%3E> <mailto:a6 at 1.com>,a6 at 1.com<a6 at 1.com%3E,a6 at 1.com>
> > <mailto:a6 at 1.com%3E,a6 at 1.com> <a6 at 1.com%3E,a6 at 1.com%3E> <
> mailto:a6 at 1.com>,88 <a6 at 1.com%3E,88>
> > <mailto:a6 at 1.com%3E,88> <a6 at 1.com%3E,88%3E>,,,,,,
> > > a7 at 1.com <mailto:a7 at 1.com> <a7 at 1.com%3E> <mailto:a7 at 1.com>,a7 at 1.com<a7 at 1.com%3E,a7 at 1.com>
> > <mailto:a7 at 1.com%3E,a7 at 1.com> <a7 at 1.com%3E,a7 at 1.com%3E> <
> mailto:a7 at 1.com>,88 <a7 at 1.com%3E,88>
> > <mailto:a7 at 1.com%3E,88> <a7 at 1.com%3E,88%3E>
> > > a8 at 1.com <mailto:a8 at 1.com> <a8 at 1.com%3E> <mailto:a8 at 1.com>,a8 at 1.com<a8 at 1.com%3E,a8 at 1.com>
> > <mailto:a8 at 1.com%3E,a8 at 1.com> <a8 at 1.com%3E,a8 at 1.com%3E> <
> mailto:a8 at 1.com>,88 <a8 at 1.com%3E,88>
> > <mailto:a8 at 1.com%3E,88> <a8 at 1.com%3E,88%3E>
> > > a9 at 1.com <mailto:a9 at 1.com> <a9 at 1.com%3E> <mailto:a9 at 1.com>,a9 at 1.com<a9 at 1.com%3E,a9 at 1.com>
> > <mailto:a9 at 1.com%3E,a9 at 1.com> <a9 at 1.com%3E,a9 at 1.com%3E> <
> mailto:a9 at 1.com>,88 <a9 at 1.com%3E,88>
> > <mailto:a9 at 1.com%3E,88> <a9 at 1.com%3E,88%3E>
> > >
> > >
> > > I need to run it thru hosted mode to be fully sure, but I strongly
> > > suspect this is the problem...
> > >
> > > Mike
> > >
> > > On Tue, Sep 14, 2010 at 5:27 AM, Kim Huang
> > <kimhuang at oit.rutgers.edu <mailto:kimhuang at oit.rutgers.edu><kimhuang at oit.rutgers.edu%3E>
> > > <mailto:kimhuang at oit.rutgers.edu>> <kimhuang at oit.rutgers.edu%3E%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E>wrote:
> > >
> > > Thomas,
> > > Please find it at attachment. BTW, following the step stated in the
> > > previous email, you can re-create the problem easily.
> > > Thanks,
> > > -Kim
> > > Thomas Amsler wrote:
> > > > Kim,
> > > >
> > > > Could you please send us your CSV file that causes this issue?
> > > >
> > > > Best,
> > > > -- Thomas
> > > >
> > > > On Mon, Sep 13, 2010 at 9:36 AM, Thomas Amsler
> > > <tpamsler at ucdavis.edu <mailto:tpamsler at ucdavis.edu><tpamsler at ucdavis.edu%3E>
> > <mailto:tpamsler at ucdavis.edu> <tpamsler at ucdavis.edu%3E> <
> mailto:tpamsler at ucdavis.edu%3E> <tpamsler at ucdavis.edu%3E%3E>
> > > > <mailto:tpamsler at ucdavis.edu>> <tpamsler at ucdavis.edu%3E%3E>
> > <mailto:tpamsler at ucdavis.edu%3E%3E> <tpamsler at ucdavis.edu%3E%3E%3E>
> > > <mailto:tpamsler at ucdavis.edu%3E%3E> <tpamsler at ucdavis.edu%3E%3E%3E>
> > <mailto:tpamsler at ucdavis.edu%3E%3E%3E><tpamsler at ucdavis.edu%3E%3E%3E%3E>wrote:
> > > >
> > > > Thank you for reporting this. We will try to verify this as soon
> > > > as possible.
> > > > -- Thomas
> > > >
> > > >
> > > > On Mon, Sep 13, 2010 at 9:03 AM, Kim Huang
> > > > <kimhuang at oit.rutgers.edu <mailto:kimhuang at oit.rutgers.edu><kimhuang at oit.rutgers.edu%3E>
> > <mailto:kimhuang at oit.rutgers.edu> <kimhuang at oit.rutgers.edu%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E> <kimhuang at oit.rutgers.edu%3E%3E>
> > > <mailto:kimhuang at oit.rutgers.edu>> <kimhuang at oit.rutgers.edu%3E%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E>
> > > <mailto:kimhuang at oit.rutgers.edu%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E%3E>wrote:
> > > >
> > > > Thomas,
> > > > Thanks for your quick response. Here you go for the new jira -
> > > > GRBK-740
> > > >
> > > > http://jira.sakaiproject.org/browse/GRBK-740
> > > >
> > > > I am willing help debug if you can point to me roughly where
> > > > you think the problem could be.
> > > >
> > > > Thanks,
> > > > -Kim
> > > > Thomas Amsler wrote:
> > > >
> > > > Kim,
> > > >
> > > > Please create a GRBK JIRA for this.
> > > >
> > > > Best,
> > > > -- Thomas
> > > >
> > > > On Mon, Sep 13, 2010 at 8:26 AM, Kim Huang
> > > > <kimhuang at oit.rutgers.edu <mailto:kimhuang at oit.rutgers.edu><kimhuang at oit.rutgers.edu%3E>
> > <mailto:kimhuang at oit.rutgers.edu> <kimhuang at oit.rutgers.edu%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E> <kimhuang at oit.rutgers.edu%3E%3E>
> > > > <mailto:kimhuang at oit.rutgers.edu> <kimhuang at oit.rutgers.edu%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E> <kimhuang at oit.rutgers.edu%3E%3E>
> > > <mailto:kimhuang at oit.rutgers.edu%3E> <kimhuang at oit.rutgers.edu%3E%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E>
> > > > <mailto:kimhuang at oit.rutgers.edu <kimhuang at oit.rutgers.edu>
> > > > <mailto:kimhuang at oit.rutgers.edu>>><kimhuang at oit.rutgers.edu%3E%3E%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E%3E>
> > > <mailto:kimhuang at oit.rutgers.edu%3E%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E%3E>
> > <mailto:kimhuang at oit.rutgers.edu%3E%3E%3E%3E><kimhuang at oit.rutgers.edu%3E%3E%3E%3E%3E>wrote:
> > > >
> > > > Hi, Folks,
> > > >
> > > > Found a serious issue of grades being truncated when
> > > > importing grades
> > > > from CSV file on Rutgers production server( GB2
> > > > -v1.2.0). Then I
> > > > reproduced it on one of the Sakai QA
> > > > servers(http://qa6-us.sakaiproject.org/portal). Please
> > > > help to confirm
> > > > whether this problem has been recorded/solved in jira.
> > > > Otherwise I
> > > > will
> > > > create a jira.
> > > >
> > > > When you import a class grades from .csv file, not all
> > > > the grades
> > > > in CSV
> > > > file are imported completed. Usually the last few (one
> > > > or two) entries
> > > > are missing. This problem only happens on .csv file,
> > > > XLS files
> > > > work just
> > > > fine.
> > > > It can be reproduced by the following steps:
> > > >
> > > > 1. create a new site, can be either project site or
> > > > course site.
> > > > 2. add about 16+ participants.
> > > > 3. add some new items in gradebook2, i.e. HW1, HW2,
> > > > HW3, etc
> > > > 4. first export the grades by clicking on
> > > > tools->export->grades &
> > > > structures->as .csv file
> > > > 5. edit the exported .csv file, added a grade to one
> > > > column for EVERY
> > > > student in the gradebook(i.e. add "88" for HW1 for 16
> > > > students, if you
> > > > have 16 students in this site.) save the file
> > > > 6. import the just saved .csv file by tools->import,
> > > > you will see from
> > > > preview data view, some grades are missing. usually the
> > > > last few
> > > > entries
> > > > are missing.
> > > >
> > > > (I investigated a little bit, the .csv file has been
> > > > read in
> > > > correctly,
> > > > which means, readInGradeDataFromImportFile in
> > > > ImportExportUtility.java
> > > > reads the complete whole set of grades from .csv file.)
> > > >
> > > > Thanks, Let me know if you have further question,
> > > >
> > > > -Kim Huang
> > > >
> > > >
> > > > _______________________________________________
> > > > gradebook2-dev mailing list
> > > > gradebook2-dev at collab.sakaiproject.org
> > <mailto:gradebook2-dev at collab.sakaiproject.org><gradebook2-dev at collab.sakaiproject.org%3E>
> > > <mailto:gradebook2-dev at collab.sakaiproject.org><gradebook2-dev at collab.sakaiproject.org%3E>
> > <mailto:gradebook2-dev at collab.sakaiproject.org%3E><gradebook2-dev at collab.sakaiproject.org%3E%3E>
> > > > <mailto:gradebook2-dev at collab.sakaiproject.org><gradebook2-dev at collab.sakaiproject.org%3E>
> > <mailto:gradebook2-dev at collab.sakaiproject.org%3E><gradebook2-dev at collab.sakaiproject.org%3E%3E>
> > > <mailto:gradebook2-dev at collab.sakaiproject.org%3E><gradebook2-dev at collab.sakaiproject.org%3E%3E>
> > <mailto:gradebook2-dev at collab.sakaiproject.org%3E%3E><gradebook2-dev at collab.sakaiproject.org%3E%3E%3E>
> > > > <mailto:gradebook2-dev at collab.sakaiproject.org<gradebook2-dev at collab.sakaiproject.org>
> > > > <mailto:gradebook2-dev at collab.sakaiproject.org>><gradebook2-dev at collab.sakaiproject.org%3E%3E>
> > <mailto:gradebook2-dev at collab.sakaiproject.org%3E%3E><gradebook2-dev at collab.sakaiproject.org%3E%3E%3E>
> > > <mailto:gradebook2-dev at collab.sakaiproject.org%3E%3E><gradebook2-dev at collab.sakaiproject.org%3E%3E%3E>
> > <mailto:gradebook2-dev at collab.sakaiproject.org%3E%3E%3E><gradebook2-dev at collab.sakaiproject.org%3E%3E%3E%3E>
> > > >
> > > >
> > > > http://collab.sakaiproject.org/mailman/listinfo/gradebook2-dev
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > [see attachment: "Gradebook(4) 08-21-52.csv", size: 562 bytes]
> > >
> > >
> > > Attachments:
> > >
> > > Gradebook(4) 08-21-52.csv
> > >
> > <
> https://smartsite.ucdavis.edu/access/content/attachment/3f65bb14-6c93-4ee0-0064-af78c775d60a/_anon_/d528c861-74ab-46b6-b0f5-eecff503b3ce/Gradebook_4_%2008-21-52.csv><https://smartsite.ucdavis.edu/access/content/attachment/3f65bb14-6c93-4ee0-0064-af78c775d60a/_anon_/d528c861-74ab-46b6-b0f5-eecff503b3ce/Gradebook_4_%2008-21-52.csv%3E>
> > <
> https://smartsite.ucdavis.edu/access/content/attachment/3f65bb14-6c93-4ee0-0064-af78c775d60a/_anon_/d528c861-74ab-46b6-b0f5-eecff503b3ce/Gradebook_4_%2008-21-52.csv%3E><https://smartsite.ucdavis.edu/access/content/attachment/3f65bb14-6c93-4ee0-0064-af78c775d60a/_anon_/d528c861-74ab-46b6-b0f5-eecff503b3ce/Gradebook_4_%2008-21-52.csv%3E%3E>
> >
> > >
> > >
> > >
> > >
> > ------------------------------------------------------------------------
> >
> > >
> > > This automatic notification message was sent by SmartSite at UCDavis
> > > (https://smartsite.ucdavis.edu/xsl-portal) from the UCD Sakai Dev
> > > site.
> > > You can modify how you receive notifications at My Workspace >
> > > Preferences.
> > >
> > >
> > >
> > >
> > > --
> > > Michael Wenk
> > > mjwenk at ucdavis.edu <mailto:mjwenk at ucdavis.edu> <mjwenk at ucdavis.edu%3E>
> > <mailto:mjwenk at ucdavis.edu> <mjwenk at ucdavis.edu%3E> <
> mailto:mjwenk at ucdavis.edu%3E> <mjwenk at ucdavis.edu%3E%3E>
> >
> > ------------------------------------------------------------------------
> >
> > This automatic notification message was sent by SmartSite at UCDavis
> > (https://smartsite.ucdavis.edu/xsl-portal) from the UCD Sakai Dev
> > site.
> > You can modify how you receive notifications at My Workspace >
> > Preferences.
> >
> >
> >
> >
> > --
> > Michael Wenk
> > mjwenk at ucdavis.edu <mailto:mjwenk at ucdavis.edu> <mjwenk at ucdavis.edu%3E>
>
> ------------------------------
>
> This automatic notification message was sent by SmartSite at UCDavis (
> https://smartsite.ucdavis.edu/xsl-portal) from the UCD Sakai Dev site.
> You can modify how you receive notifications at My Workspace > Preferences.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/gradebook2-dev/attachments/20100914/37d9099b/attachment-0001.html