Single call to fgetcsv() exhausting large memory limit in PHP - memory

I am trying to read two lines from a average-sized CSV (10 MB), the header and the first line. I have the memory limit for PHP set to 64 MB. Yet, on the second call to fgetcsv(), I get a memory exhausted error.
I've removed every unessential part of the script to debug, and traced memory usage step-by-step, and I still can't figure it out. Right after the first call, memory usage is under 1 MB, but at the second call, the memory exhaust error happens.
Here's the code:
<?php
function trace($msg) {
echo "<br>" . date("Y-m-d H:i:s") . " - " . memory_get_usage() . " bytes - {$msg}";
}
ini_set('display_errors', 'on');
error_reporting(E_ALL);
ob_implicit_flush(true);
trace("CSV Memory Limit test");
$path = "../data/uploaded/1405712684_base_leitores_july2014.csv";
trace("path = {$path}");
$fp = fopen($path, "r");
trace("file pointer opened");
trace("getting header...");
$header = fgetcsv($fp, 1024);
trace("header fetched");
var_dump($header);
trace("header displayed");
trace("fetching another line");
$l1 = fgetcsv($fp, 1024);
trace("line fetched");
var_dump($header);
trace("line displayed");
Here's the output:
2014-07-18 16:06:26 - 249544 bytes - CSV Memory Limit test
2014-07-18 16:06:26 - 249800 bytes - path = ../data/uploaded/1405712684_base_leitores_july2014.csv
2014-07-18 16:06:26 - 250840 bytes - file pointer opened
2014-07-18 16:06:26 - 250840 bytes - getting header...
2014-07-18 16:06:26 - 279280 bytes - header fetched
array (size=111)
0 => string 'razao social' (length=12)
1 => string 'Identificaao' (length=13)
2 => string 'save' (length=4)
3 => string 'id' (length=2)
4 => string 'categoria' (length=9)
5 => string 'nome fantasia' (length=13)
6 => string 'email' (length=5)
7 => string 'website' (length=7)
8 => string 'logradouro' (length=10)
9 => string 'endereco' (length=8)
10 => string 'numero' (length=6)
11 => string 'complemento' (length=11)
12 => string 'bairro' (length=6)
13 => string 'cidade' (length=6)
14 => string 'estado' (length=6)
15 => string 'cep' (length=3)
16 => string 'localidade' (length=10)
17 => string 'ddd' (length=3)
18 => string 'telefone' (length=8)
19 => string 'fax' (length=3)
20 => string 'contato' (length=7)
21 => string 'cargo' (length=5)
22 => string 'departamento' (length=12)
23 => string 'ramo de atividade' (length=17)
24 => string 'data de nascimento' (length=18)
25 => string 'especialidade' (length=13)
26 => string 'linha de atuacao' (length=16)
27 => string 'data de fundacao' (length=16)
28 => string 'classificacao de leitor' (length=23)
29 => string 'observacoes' (length=11)
30 => string 'is_protocolada' (length=14)
31 => string 'is_assinante' (length=12)
32 => string 'qtd_reclamacoes' (length=15)
33 => string 'error' (length=5)
34 => string 'errorReport' (length=11)
35 => string 'saved' (length=5)
36 => string 'onTrash' (length=7)
37 => string 'SugestAo de matEria' (length=19)
38 => string 'Campo2
#redacted#' (length=31)
39 => string '1' (length=1)
40 => string '0' (length=1)
41 => string '141657' (length=6)
42 => string '2000000041' (length=10)
43 => string '' (length=0)
44 => string '' (length=0)
45 => string '' (length=0)
46 => string 'RUA' (length=3)
47 => string '#redacted#' (length=15)
48 => string '#redacted#' (length=3)
49 => string '' (length=0)
50 => string 'JABAQUARA' (length=9)
51 => string 'PARI' (length=4)
52 => string 'SP' (length=2)
53 => string '#redacted#' (length=7)
54 => string '' (length=0)
55 => string '11' (length=2)
56 => string '#redacted#' (length=9)
57 => string '#redacted#' (length=9)
58 => string '#redacted#' (length=5)
59 => string '#redacted#' (length=12)
60 => string '' (length=0)
61 => string 'Centros Automotivos, Mec‰nicas e Oficinas - Leves' (length=49)
62 => string '' (length=0)
63 => string '' (length=0)
64 => string '' (length=0)
65 => string '' (length=0)
66 => string '' (length=0)
67 => string 'Centros Automotivos, MecAnicas e Oficinas - Leves ' (length=50)
68 => string 'N' (length=1)
69 => string '0' (length=1)
70 => string '0' (length=1)
71 => string '0' (length=1)
72 => string '' (length=0)
73 => string '1' (length=1)
74 => string '0' (length=1)
75 => string '' (length=0)
76 => string '
#redacted#' (length=16)
77 => string '2' (length=1)
78 => string '0' (length=1)
79 => string '228109' (length=6)
80 => string '2000000041' (length=10)
81 => string '0800 AUTOMOTIVE' (length=15)
82 => string '#redacted##hotmail.com' (length=32)
83 => string '' (length=0)
84 => string 'RUA' (length=3)
85 => string '#redacted#' (length=7)
86 => string '322' (length=3)
87 => string '' (length=0)
88 => string '' (length=0)
89 => string 'SAO PAULO' (length=9)
90 => string 'SP' (length=2)
91 => string '#redacted#' (length=7)
92 => string '' (length=0)
93 => string '11' (length=2)
94 => string '#redacted#' (length=9)
95 => string '#redacted#' (length=9)
96 => string '#redacted#' (length=40)
97 => string 'PROPRIETARIO' (length=12)
98 => string '' (length=0)
99 => string 'Centros Automotivos, Mec‰nicas e Oficinas - Leves' (length=49)
100 => string '' (length=0)
101 => string '' (length=0)
102 => string '' (length=0)
103 => string '' (length=0)
104 => string '' (length=0)
105 => string 'Centros Automotivos, MecAnicas e Oficinas - Leves ' (length=50)
106 => string '' (length=0)
107 => string '0' (length=1)
108 => string '0' (length=1)
109 => string '0' (length=1)
110 => string 'Registro criado' (length=15)
2014-07-18 16:06:26 - 287608 bytes - header displayed
2014-07-18 16:06:26 - 287608 bytes - fetching another line
( ! ) Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 72 bytes) in /home/lqdi/quantum.pranaeditora.com.br/sandbox/test_csv_read.php on line 25
Call Stack
# Time Memory Function Location
1 0.0011 249048 {main}( ) ../test_csv_read.php:0
2 0.0093 287520 fgetcsv ( ) ../test_csv_read.php:25
I am running php5-fpm (5.5.9) and nginx (1.4.6) on Ubuntu 14.04 LTS.

Finally, after a lot of headbanging, I found the problem: line endings.
Somewhy, Mac's Excel was exporting a CSV with a \r\n on the header and \n on all other lines. The header would come in nicely, but fgetcsv understood the second line was the entire document.
The fix was to add the following before fopen:
ini_set('auto_detect_line_endings',TRUE);
This will make fgetcsv detect line endings corrently when parsing the CSV file.

Related

Rails group_by id and group created

I have a model call GcMission
it contains gamecharacter_id, mission_id and of course created_at.
Now I need to group_by gamecharacter_id, and then group the created_at to see how many mission_id per day.
I assum the result will be like
{
69 => {
'2017-08-18' => 1
},
75 => {
'2017-08-18' => 2
},
78 => {
'2017-08-18' => 1,
'2017-08-19' => 1,
}
}
But I am stuck at that.
Here is what I've written.
GcMission.select('gamecharacter_id, mission_id, created_at').where(mission_id: assign_mission_ids).where('finish_counter >= 1').group_by{|a| a.gamecharacter_id}
And the result is like
{
69 => [
[0] #<GcMission:0x007ff6c6a53650> {
:gamecharacter_id => 69,
:mission_id => 3,
:created_at => Tue, 18 Apr 2017 03:20:36 UTC +00:00
}
],
75 => [
[0] #<GcMission:0x007ff6c6a52d18> {
:gamecharacter_id => 75,
:mission_id => 3,
:created_at => Tue, 18 Apr 2017 06:38:27 UTC +00:00
},
[1] #<GcMission:0x007ff6c6a52408> {
:gamecharacter_id => 75,
:mission_id => 4,
:created_at => Tue, 18 Apr 2017 07:55:40 UTC +00:00
}
],
78 => [
[0] #<GcMission:0x007ff6c6a51be8> {
:gamecharacter_id => 78,
:mission_id => 3,
:created_at => Tue, 18 Apr 2017 17:29:24 UTC +00:00
},
[1] #<GcMission:0x007ff6c6a51580> {
:gamecharacter_id => 78,
:mission_id => 4,
:created_at => Wed, 19 Apr 2017 03:20:31 UTC +00:00
}
]
}
Try Following
gc_missions = GcMission.select("gamecharacter_id, DATE(created_at) AS date,
COUNT(mission_id) AS mission_count")
.group('gamecharacter_id, date')
If you want count of unique mission_id per gamecharacter_id per day use
COUNT(DISTINCT(mission_id))
You may like to verify using following code
gc_missions.each {|gc| puts "#{gc.gamecharacter_id}, #{gc.date}, #{gc.mission_count}"}

Google API 401 Invalid Credentials

I'm trying to use Oauth2 to access Analytics data. I successfully get authorization code from and change it for authorization token. But when I try to get any data from Analytics API I get 401 Invalid Credentials error.
The code I'm using is from PHP client library:
<?php
ini_set('display_errors', 1);
error_reporting(E_ALL);
require_once 'lib/API/Google_Client.php';
require_once 'lib/API/contrib/Google_AnalyticsService.php';
session_start();
$client = new Google_Client();
$client->setApplicationName("APP name");
$client->setClientId('xxx.apps.googleusercontent.com');
$client->setClientSecret('mysecret');
$client->setRedirectUri('http://example.com');
$client->setDeveloperKey('mykey');
$client->setScopes('https://www.googleapis.com/auth/analytics');
$service = new Google_AnalyticsService($client);
if (isset($_GET['logout'])) {
session_destroy();
}
if (isset($_GET['code'])) {
$client->authenticate();
$_SESSION['token'] = $client->getAccessToken();
$redirect = 'http://' . $_SERVER['HTTP_HOST'] . $_SERVER['PHP_SELF'];
header('Location: ' . filter_var($redirect, FILTER_SANITIZE_URL));
}
if (isset($_SESSION['token'])) {
$client->setAccessToken($_SESSION['token']);
// $client->refreshToken('refresh-token');
print_r(json_decode($_SESSION['token']));
if ($client->isAccessTokenExpired()) {
session_destroy();
header('Location: http://example.com');
}
}
if ($client->getAccessToken()) {
$props = $service->management_webproperties->listManagementWebproperties("~all");
print "<h1>Web Properties</h1><pre>" . print_r($props, true) . "</pre>";
$accounts = $service->management_accounts->listManagementAccounts();
print "<h1>Accounts</h1><pre>" . print_r($accounts, true) . "</pre>";
$segments = $service->management_segments->listManagementSegments();
print "<h1>Segments</h1><pre>" . print_r($segments, true) . "</pre>";
$goals = $service->management_goals->listManagementGoals("~all", "~all", "~all");
print "<h1>Segments</h1><pre>" . print_r($goals, true) . "</pre>";
$_SESSION['token'] = $client->getAccessToken();
} else {
$authUrl = $client->createAuthUrl();
print "<a class='login' href='$authUrl'>Connect Me!</a>";
}
?>
I tripplechecked my credentials. My access token looks something like this:
Object
(
[access_token] => ya29.blablabla
[token_type] => Bearer
[expires_in] => 3600
[refresh_token] => 1/Eblablabla
[created] => 1413873721
)
When I try to get webproperties from GA account I get 401 error:
Google_ServiceException Object
(
[errors:protected] => Array
(
[0] => Array
(
[domain] => global
[reason] => authError
[message] => Invalid Credentials
[locationType] => header
[location] => Authorization
)
)
[message:protected] => Error calling GET https://www.googleapis.com/analytics/v3/management/accounts/~all/webproperties?key=AIzaSyCDOiczVHVPiXhynaTkLkQ-Jp2Kr8kngS0: (401) Invalid Credentials
[string:Exception:private] =>
[code:protected] => 401
[file:protected] => ...lib//API/io/Google_REST.php
[line:protected] => 66
[trace:Exception:private] => Array
(
[0] => Array
(
[file] => .../lib/API/io/Google_REST.php
[line] => 36
[function] => decodeHttpResponse
[class] => Google_REST
[type] => ::
[args] => Array
(
[0] => Google_HttpRequest Object
(
[batchHeaders:Google_HttpRequest:private] => Array
(
[Content-Type] => application/http
[Content-Transfer-Encoding] => binary
[MIME-Version] => 1.0
[Content-Length] =>
)
[url:protected] => https://www.googleapis.com/analytics/v3/management/accounts/~all/webproperties?key=AIzaSyCDOiczVHVPiXhynaTkLkQ-Jp2Kr8kngS0
[requestMethod:protected] => GET
[requestHeaders:protected] => Array
(
[authorization] => Bearer ya29.xxx
)
[postBody:protected] =>
[userAgent:protected] => APP name google-api-php-client/0.6.5
[responseHttpCode:protected] => 401
[responseHeaders:protected] => Array
(
[vary] => Origin
Referer
X-Origin
[www-authenticate] => Bearer realm="https://accounts.google.com/AuthSubRequest", error=invalid_token
[content-type] => application/json; charset=UTF-8
[date] => Tue, 21 Oct 2014 14:00:46 GMT
[expires] => Tue, 21 Oct 2014 14:00:46 GMT
[cache-control] => private, max-age=0
[x-content-type-options] => nosniff
[x-frame-options] => SAMEORIGIN
[x-xss-protection] => 1; mode=block
[server] => GSE
[alternate-protocol] => 443:quic,p=0.01
[transfer-encoding] => chunked
)
[responseBody:protected] => {"error":{"errors":[{"domain":"global","reason":"authError","message":"Invalid Credentials","locationType":"header","location":"Authorization"}],"code":401,"message":"Invalid Credentials"}}
[accessKey] =>
)
)
)
[1] => Array
(
[file] => .../lib/API/service/Google_ServiceResource.php
[line] => 186
[function] => execute
[class] => Google_REST
[type] => ::
[args] => Array
(
[0] => Google_HttpRequest Object
(
[batchHeaders:Google_HttpRequest:private] => Array
(
[Content-Type] => application/http
[Content-Transfer-Encoding] => binary
[MIME-Version] => 1.0
[Content-Length] =>
)
[url:protected] => https://www.googleapis.com/analytics/v3/management/accounts/~all/webproperties?key=AIzaSyCDOiczVHVPiXhynaTkLkQ-Jp2Kr8kngS0
[requestMethod:protected] => GET
[requestHeaders:protected] => Array
(
[authorization] => Bearer ya29.xxx
)
[postBody:protected] =>
[userAgent:protected] => APP name google-api-php-client/0.6.5
[responseHttpCode:protected] => 401
[responseHeaders:protected] => Array
(
[vary] => Origin
Referer
X-Origin
[www-authenticate] => Bearer realm="https://accounts.google.com/AuthSubRequest", error=invalid_token
[content-type] => application/json; charset=UTF-8
[date] => Tue, 21 Oct 2014 14:00:46 GMT
[expires] => Tue, 21 Oct 2014 14:00:46 GMT
[cache-control] => private, max-age=0
[x-content-type-options] => nosniff
[x-frame-options] => SAMEORIGIN
[x-xss-protection] => 1; mode=block
[server] => GSE
[alternate-protocol] => 443:quic,p=0.01
[transfer-encoding] => chunked
)
[responseBody:protected] => {"error":{"errors":[{"domain":"global","reason":"authError","message":"Invalid Credentials","locationType":"header","location":"Authorization"}],"code":401,"message":"Invalid Credentials"}}
[accessKey] =>
)
)
)
[2] => Array
(
[file] => .../lib/API/contrib/Google_AnalyticsService.php
[line] => 1010
[function] => __call
[class] => Google_ServiceResource
[type] => ->
[args] => Array
(
[0] => list
[1] => Array
(
[0] => Array
(
[accountId] => ~all
)
)
)
)
[3] => Array
(
[file] => .../oatuhCallback.php
[line] => 46
[function] => listManagementWebproperties
[class] => Google_ManagementWebpropertiesServiceResource
[type] => ->
[args] => Array
(
[0] => ~all
)
)
)
[previous:Exception:private] =>
)
What am I doing wrong?
I found answer myself.
When using Google API you ALWAYS have to create Public access key which gives you developer key. That is your right developer key. I followed instructions from some forum and got wrong directions for getting my developer key.
Unfortunately this is not mentioned in Googles OAuth integration instructions.

Highcharts no display even when there's data

As you can see the chart displays with yaxis and axis but the value for xaxis is not shown. My data looks like this,
Array
(
[series] => Array
(
[0] => Array
(
[answer_option_group_id] => 0
[name] => All
[data] => Array
(
[999] => 1
[172] => 1
[173] => 0
[174] => 77
[194] => 6
)
)
[174] => Array
(
[answer_option_group_id] => 174
[name] => Does Not Meet
[data] => Array
(
[1459dcb35a7888ab4ff3dd89c07d679e] => 7
[c424759c02fd0900e3ec7c58c745138c] => 1
[e789fa7486d59950d382b8c395de166a] => 2
[b6c632c0a07b9feaa1eefe547a1a6aa8] => 4
[2ecfa0fc3d3990a2ff30009bea9b8ae0] => 2
[b6a52c346ad6a9f4cc58ddfe3c83939e] => 8
[36894aa1fcc16e360813d1ef430bb8fa] => 6
[4284ce3f8523823a457602fe5997fcfa] => 5
[e45e137a7da3e3fe5dc3785d7d628e44] => 11
[fb9bd545acc9397e0c84a8ff5d1a3b92] => 3
[1bcace373da4bb1e821158e3a5fa80a4] => 7
[88928dea322b243c18082e23aa9bc7a4] => 1
[2e9cf454d6eb69e69e744fcc15287f49] => 3
[eaa42f3a7bf5e5f2ac3198d5f8d487a4] => 3
[690463148302dc63a0dfd327ab5253f9] => 1
[325795fe2d5f5a22136b2362ee863be4] => 4
[18bb150bdbf978e058b44ba52d836591] => 1
[418702f6d334d6531e9ac4883a690adb] => 7
[c1ebb58ebb59c5bc6bb87a567c763cfe] => 1
)
)
)
)
I used console.log() to check if I got the data right and the result looked like this,
["All", "Does Not Meet"]
[85, 77]
I tried this on having a jsfiddle and it worked but on my server it didn't. I also dont think this is about the including of scripts since it already displays the chart only that the values are not shown. Any help would be appreciated. Thanks.

How would I filter out sub documents?

1) How can I grab all songs with more than 50 likes? The result set should return an array of songs instead of the whole document.
2) How can I return the results of a specific song id instead of the whole document?
So far I've come across this concept of virtual collections for embedded objects. However, it hasn't been implemented since. Although, on the same link, someone suggests using the aggregation framework for such a thing. Is this a viable solution?
I have the following document structure:
document = {
'date' => Time.now.utc.strftime('%Y%m%d'),
'songs' => {
'1' => {
'views' => {
'non_uniques' => 2000,
'uniques' => 203
},
'countries' => {
'us' => 123,
'uk' => 11
},
'likes' => 123,
'comments' => 123
},
'2' => {
'views' => {
'non_uniques' => 2000,
'uniques' => 203
},
'countries' => {
'us' => 123,
'uk' => 11
},
'likes' => 123,
'comments' => 123
}
},
'sign_ins' => {
'non_uniques' => 2000,
'uniques' => 203
}
}
Expected output (as requested)
For question 1), somewhere within the lines of:
['1' => {
'views' => {
'non_uniques' => 2000,
'uniques' => 203
},
'countries' => {
'us' => 123,
'uk' => 11
},
'likes' => 123,
'comments' => 123
},
'2' => {
'views' => {
'non_uniques' => 2000,
'uniques' => 203
},
'countries' => {
'us' => 123,
'uk' => 11
},
'likes' => 123,
'comments' => 123
}]
For 2):
'1' => {
'views' => {
'non_uniques' => 2000,
'uniques' => 203
},
'countries' => {
'us' => 123,
'uk' => 11
},
'likes' => 123,
'comments' => 123
}
Use aggregation framework like this:
db.songs.aggregate({$unwind:"$songs"},{$match:{"songs.likes":{$gt:50}}})
Now you can add other stages like {$group} if you want things back in one doc, or $project to change the field names, etc.
Change the '=>' delimiter in the document. Then you will have a JSON file
document = JSON.parse(document)
You will get a hash. Then check if likes > 50 in the hash
document[].each do |song|
song[].each do |id|
if id[likes].to_i > 50
#Do what you want to do
end
end
end
Hope that helps
Try:
songs_arr = []
document["songs"].keys.each do |key|
songs_arr << "insert songs here" if document["songs"][key]["likes"] > 50
end

Changing the validator of an embedded form field

I have this schema:
propel:
autor:
id: ~
nombre: { type: varchar, size: 255, required: true }
libro:
id: ~
autor_id: { type: integer, size: 11, foreignTable: autor,
foreignReference: id}
titulo: { type: varchar, size: 255 }
paginas: { type: varchar, size: 255, required: true }
and this form class:
class AutorForm extends BaseAutorForm
{
public function configure()
{
$this->embedRelation('Libro');
}
public function foo()
{
$this->validatorSchema['Libro']['newLibro1']['paginas'] = new
sfValidatorPass();
return $this;
}
}
I'm calling foo() after the bind() (inside processForm()).
After submiting the Auto-Libro form, if I don't insert anything in the
field 'paginas' of the first embedded form (Libro), it shows
"Required".
But.. why if paginas has a validator-pass?
EDIT: after matt's answer, this is my code:
var_dump($this->embeddedForms['Libro']->validatorSchema['newLibro1']['paginas']);
$this->embeddedForms['Libro']->validatorSchema['newLibro1']['paginas'] = new sfValidatorPass(array('required' => false));
var_dump($this->embeddedForms['Libro']->validatorSchema['newLibro1']['paginas']);
It prints this:
object(sfValidatorString)[152]
protected 'requiredOptions' =>
array
empty
protected 'defaultMessages' =>
array
'required' => string 'Required.' (length=9)
'invalid' => string 'Invalid.' (length=8)
'max_length' => string '"%value%" is too long (%max_length% characters max).' (length=52)
'min_length' => string '"%value%" is too short (%min_length% characters min).' (length=53)
protected 'defaultOptions' =>
array
'required' => boolean true
'trim' => boolean false
'empty_value' => string '' (length=0)
'max_length' => null
'min_length' => null
protected 'messages' =>
array
'required' => string 'Required.' (length=9)
'invalid' => string 'Invalid.' (length=8)
'max_length' => string '"%value%" is too long (%max_length% characters max).' (length=52)
'min_length' => string '"%value%" is too short (%min_length% characters min).' (length=53)
protected 'options' =>
array
'required' => boolean true
'trim' => boolean false
'empty_value' => string '' (length=0)
'max_length' => int 255
'min_length' => null
object(sfValidatorPass)[196]
protected 'requiredOptions' =>
array
empty
protected 'defaultMessages' =>
array
'required' => string 'Required.' (length=9)
'invalid' => string 'Invalid.' (length=8)
protected 'defaultOptions' =>
array
'required' => boolean true
'trim' => boolean false
'empty_value' => null
protected 'messages' =>
array
'required' => string 'Required.' (length=9)
'invalid' => string 'Invalid.' (length=8)
protected 'options' =>
array
'required' => boolean false <<<<<<<<<<<< FALSE <<<<<<<<<<<<<<<<<<
'trim' => boolean false
'empty_value' => null
The problem is still the same: when i try to submit the form, the field 'paginas' is still required. Why?
sf 1.4/propel 1.6
Javi
Almost all validators, including the sfValidatorPass, default to required=true.
I believe the sfValidatorPass doesn't mean "it always passes validation" it means "pass the submitted value through without cleaning it or checking anything."
sfValidatorPass is an identity validator. It simply returns the value unmodified.
So I think you can just do
$this->validatorSchema['Libro']['newLibro1']['paginas'] = new
sfValidatorPass(array('required' => false));

Resources