We always hesitate to ask Dyalog for an enhancement, for fear that while we think it is important at the time, in the long run it will turn out to be a mistake or not needed. We have users too, and we see this often from the supplier side. People often ask for things they really don't need.

When ⎕CSV was being designed, we insisted on the invert variant for character columns to be imported as matrices rather than vectors of vectors. We wanted this because nesting is expensive, and takes up a lot of space. And because our custom DBMS is optimized for fixed-width character columns. This turned out to be a mistake.

Dysfunctional CSV files are the rule, not the exception. It's easy to find examples of files with columns that are mostly empty, but have a few entries thousands of characters wide. If you run into a large file with such a column, it's game over (WSFULL) if you use the invert variant. Throw in a few characters of ⎕DR 160 and the problem is even worse.

If the CSV file is small, nesting is not an issue. If the CSV is large, it must be read in blocks and there is no way to know what the widest value will be until you have read the entire file.

Thus, it is better to always read a block of a column as a nested vector, inspect the length of each value, and only invert the column block if the matrix form is smaller than the nested form. In addition we can specify am optional column width limit to simply truncate any wide data:

ProcessColumn←{
     w←⌈/≢¨⍵
     m←⍺.ColumnWidthLimit
     (m>0)∧w≤m:↑⍵
     (m>0)∧w>m:↑m↑¨⍵
     (w×≢⍵)<⎕SIZE'⍵':↑⍵
     ⍵
 }

If, in addition, we store the size of the nested column block and the maximum width, the most efficient space representation of the column can be determined at the end.

We over-complicated the design of ⎕CSV for no good reason.