Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [csv]

34 questions
50
votes
4 answers

Should UTF-8 CSV files contain a BOM (byte order mark)?

Our line-of-business software allows the user to save certain data as CSV. Since there are a lot of different formats (all called "CSV") in use in the wild, we are tying to decide what the "default format" should look like. Regarding line/field…
Heinzi
  • 9,868
28
votes
5 answers

Can the csv format be defined by a regex?

A colleague and I have recently argued over whether a pure regex is capable of fully encapsulating the csv format, such that it is capable of parsing all files with any given escape char, quote char, and separator char. The regex need not be…
Spencer Rathbun
  • 3,576
22
votes
6 answers

Is CSV a good alternative to XML and JSON?

Is CSV considered a good option against XML and JSON for programming languages? I generally use XML and JSON (or sometimes a plain text file) as flat file storage. However, recently I came across an CSV implementation in PHP. I generally have seen…
Vishwas
  • 1,891
14
votes
5 answers

unit tests for a csv parser

What tests should I use to unit test a CSV parser? I have a simple CSV parser in C#, and I want to be sure that I have good unit test coverage of all the common (and uncommon) edge cases. What tests should I use to identify potential problems and…
Joel Coehoorn
  • 1,945
8
votes
4 answers

CSV file generation using Java

We have a requirement in our project to generate a big CSV file every 2 hours using a Java program. This file will have around 60,000 lines (around 120 characters per line). I am not sure about the size yet. I would like to know if I would run into…
java_mouse
  • 2,657
  • 17
  • 23
6
votes
1 answer

What to do when test data takes up most of repo size?

TL;DR: Given a repo with a lot of big fixture tests, when should I look for a solution where the golden files are not part of the repo anymore? Where should they be stored? The setup Testing a complex data processing project with very diverse input…
Turion
  • 169
5
votes
2 answers

Multi-level validation in C#

I have a Console project reads inputs from CSV file and tries to save them to database. For that, I created a class Person that maps a CSV row. The CSV file has two columns Name and Age. Person class is like. class Person { public string Name; …
Mhd
  • 195
  • 1
  • 7
5
votes
4 answers

Best way of validating Class properties

Background: I have a CSV file, which I need to ready and validate each element in each row and create a collection of a class, having valid data. i.e CSV File looks like: EmpID,FirstName,LastName,Salary 1,James,Help,100000 …
user2697452
  • 153
  • 1
  • 1
  • 3
4
votes
1 answer

When do regression models outperform naive methods?

Case1: I have the following task to do: Training by the consecutive 3 days to predict the each 4th day. Each day data represents one CSV file which has dimension 24x25. Every datapoints of each CSV file are pixels. Now, I need to do that, predict…
S. M.
  • 37
  • 11
4
votes
5 answers

The best way to handle exceptions?

I have the following method, which needs to return a List, but exceptions might occur along the way, and I don't want to handle them in my application, mainly because I don't even know how to handle them. The idea is simple: read data from a CSV…
MasterTJ123
  • 75
4
votes
5 answers

Import large csv files

I've been tasked to query 2 large csv files which are roughly 1 GB in size each. The files contain related data, so file one may contain a list of order numbers, Order Dates, etc and the file may contain the order lines, stock codes, quantity,…
Rhodes73
  • 43
3
votes
1 answer

Meaningful response to the user after his uploaded CSV was processed?

A user uploads a CSV to the server to be inserted into the database and I would like to return a meaningful response to this user. My thoughts : { inserted : x records discarded : y records } I believe also that I should specify what records…
Oleg
  • 181
2
votes
8 answers

Fail fast is brittle

I am creating a CSV consumer (with Java). There is one field / column that should contain one of the values "Rename" or "Move". I implemented this by allowing mixed case of letters, and blank space at either end. To be precise I used…
NimChimpsky
  • 4,670
2
votes
3 answers

Back-end solution for pulling from CSV files

I'm building a data visualization that displays COVID information for the United States, at the city, state, and county level. The ultimate source of truth are three CSVs published by the New York Times on Github in this…
InspectorDanno
  • 137
2
votes
2 answers

SQL query or C# .net code for csv files import?

My aim here is to find out the best possible and feasible solution for my dilemma. I want to import some csv file (may contain around 50~60K records) into database after some manipulation into the files, manipulation includes comparison such as one…
Shilpa Soni
  • 121
  • 5
1
2 3