I am facing a problem trying to run a fairly large optimization problem in my opinion. You can see the code below. The variable b size is 500 x 96. What I am trying to do is to match a sum of timeseries profiles (351236 x 15 min timesteps) with a bigger profile by minimizing their difference. With the same formulation and a much smaller problem (672 timesteps and a b variable of the size 10 x 5) the problem is solved in under 2 seconds without a problem. But when I am running it for the full scale problem I get the error you see below.
I am running this on Jupyter Lab and python 3.7.4. The python installation is done with conda.
I would expect the problem to solve as with the much smaller problem. But when I run this one, RAM usage explodes up to 100 GB (about 99% of the available RAM on the server). After a while the RAM usage goes down and then a periodical swinging begins (RAM goes up and down from 50% to 100% every few minutes). From the error and after a lot of googling my suspicion is that the problem is too big for the memory and that at some point data is getting broken down to smaller pieces. I do not think it reaches to the point, where the solver does its work. I tried to optimize the code by vectorizing everything (current version) and trying not to have loops etc. in the formulation. But this did not change anything. Do you guys have any clue if this is a bug or a limitation? Or do you maybe have an idea on how to solve this?
X_opt = cp.Constant(np.asarray(X.iloc[:,:500])) # the array size is (35136,500)
K_opt = cp.Constant(np.asarray(K.YearlyDemand)) # the vector size is 96
b = cp.Variable((500,96),boolean = True, value = np.zeros((500,96)))
Y_opt = cp.Constant(np.asarray(y)) # the vector size is 35136
constraints = []
constraints.append( cp.sum(b, axis = 0) == 1 ) # the sum of the elements of every column of b must be equal to 1
constraints.append( cp.sum(b, axis = 1) <= 1 ) # the sum of the elements of every row of b must be smaller or equal to 1
objective = cp.Minimize(cp.sum(cp.abs(Y_opt-cp.sum((cp.diag(K_opt)*((X_opt#b).T)).T, axis = 1))))
prob = cp.Problem(objective, constraints)
prob.solve(solver = cp.GLPK_MI, verbose = True)
ValueError Traceback (most recent call last)
in
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\problems\problem.py in solve(self, *args, **kwargs)
287 else:
288 solve_func = Problem._solve
--> 289 return solve_func(self, *args, **kwargs)
290
291 #classmethod
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\problems\problem.py in _solve(self, solver, warm_start, verbose, parallel, gp, qcp, **kwargs)
567 self._construct_chains(solver=solver, gp=gp)
568 data, solving_inverse_data = self._solving_chain.apply(
--> 569 self._intermediate_problem)
570 solution = self._solving_chain.solve_via_data(
571 self, data, warm_start, verbose, kwargs)
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\reductions\chain.py in apply(self, problem)
63 inverse_data = []
64 for r in self.reductions:
---> 65 problem, inv = r.apply(problem)
66 inverse_data.append(inv)
67 return problem, inverse_data
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\reductions\matrix_stuffing.py in apply(self, problem)
98 # Batch expressions together, then split apart.
99 expr_list = [arg for c in cons for arg in c.args]
--> 100 Afull, bfull = extractor.affine(expr_list)
101 if 0 not in Afull.shape and 0 not in bfull.shape:
102 Afull = cvxtypes.constant()(Afull)
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\utilities\coeff_extractor.py in affine(self, expr)
76 size = sum([e.size for e in expr_list])
77 op_list = [e.canonical_form[0] for e in expr_list]
---> 78 V, I, J, b = canonInterface.get_problem_matrix(op_list, self.id_map)
79 A = sp.csr_matrix((V, (I, J)), shape=(size, self.N))
80 return A, b.flatten()
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\cvxcore\python\canonInterface.py in get_problem_matrix(linOps, id_to_col, constr_offsets)
65
66 # Unpacking
---> 67 V = problemData.getV(len(problemData.V))
68 I = problemData.getI(len(problemData.I))
69 J = problemData.getJ(len(problemData.J))
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\cvxcore\python\cvxcore.py in getV(self, values)
320
321 def getV(self, values):
--> 322 return _cvxcore.ProblemData_getV(self, values)
323
324 def getI(self, values):
ValueError: negative dimensions are not allowed
This problem is solved here:
https://github.com/cvxgrp/cvxpy/issues/826#issuecomment-648618636
Note that the general problem is that large problems create underlying matrixies too large for the numpy.int32 which CVXPY uses. You can modify the code in CVXPY fairly easily to continue using the SCS solver.
You will have to modify the file canonInterface.py here:
D:\Anaconda3\envs\py37DuAL\lib\site-packages\cvxpy\cvxcore\python\
If you have trouble finding the second file to modify, just modify the first one, and use the traceback to find the second file.
Related
I ran jupyter through docker run -p 8888:8888 jupyter/scipy-notebook.
from sklearn.datasets import make_classification
X, y = make_classification(10000,2000)
causes kernel to die, while X, y = make_classification(100000,2000) gives
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
Cell In [1], line 3
1 from sklearn.datasets import make_classification
----> 3 X, y = make_classification(100000,2000)
File /opt/conda/lib/python3.10/site-packages/sklearn/datasets/_samples_generator.py:220, in make_classification(n_samples, n_features, n_informative, n_redundant, n_repeated, n_classes, n_clusters_per_class, weights, flip_y, class_sep, hypercube, shift, scale, shuffle, random_state)
217 n_samples_per_cluster[i % n_clusters] += 1
219 # Initialize X and y
--> 220 X = np.zeros((n_samples, n_features))
221 y = np.zeros(n_samples, dtype=int)
223 # Build the polytope whose vertices become cluster centroids
MemoryError: Unable to allocate 1.49 GiB for an array with shape (100000, 2000) and data type float64
It looks like for the larger (100000,2000 ) data size, allocation failed and errored out early.
Did the smaller (10000,2000 ) data size allocate successfully
since there's no error raised?
So what is killing the kernel and why is the initial memory allocation insufficient as a check? It looks like there is extra memory allocated as the code runs?
When running the 10000, 2000 code, docker stats shows
MEM USAGE / LIMIT beginning at 152.2MiB / 966.2MiB then shooting to 369.2MiB / 966.2MiB, waiting for a few seconds, then kernel died pop-up.
369 looks very far from 966 available, did the MEM USAGE suddenly jump from 369 to above 966 in 1 step?
If the kernel indeed died because MEM USAGE was too high, how do I know when (eg. what % of available memory usage) to start being careful and deleting unused variables?
I am replicating ResNet (source: https://arxiv.org/abs/1512.03385).
I ran into the error "TypeError: '<' not supported between instances of 'Tensor' and 'list'" when trying to go through several different dataset in different sections of my code.
I tried different fixes but none worked: (i) I deleted enumerate cause I worried that using this may cause the problem (ii) I tried to go through dataloader rather than dataset but it didn't work
1st time: When I tried to view images:
for images, _ in train_loader:
print('images.shape:', images.shape)
plt.figure(figsize=(16,8))
plt.axis('off')
plt.imshow(torchvision.utils.make_grid(images, nrow=16).permute((1, 2, 0)))
break
2nd/3rd time: when I tried to validate/test the resnet:
with torch.no_grad():
for j, inputs, labels in enumerate(test_loader, start=0):
outputs = resnet_models[i](inputs)
_, prediction = torch.max(outputs, dim=1)
You may notice that I didn't run into this error when training the resnet, and the code is quite similar:
for batch, data in enumerate(train_dataloader, start=0):
inputs, labels = data
inputs, labels = inputs.to(device), labels.to(device)
Error message (taking the first error as an example. The rest is pretty much the same)
TypeError Traceback (most recent call last)
Input In [38], in <cell line: 8>()
6 print("Images AFTER NORMALIZATION")
7 print("--------------------------")
----> 8 for images, _ in training_data:
9 sort=False
10 print('images.shape:', images.shape)
File ~/miniconda3/envs/resnet/lib/python3.9/site->packages/torch/utils/data/dataset.py:471, in Subset.getitem(self, idx)
469 if isinstance(idx, list):
470 return self.dataset[[self.indices[i] for i in idx]]
--> 471 return self.dataset[self.indices[idx]]
File ~/miniconda3/envs/resnet/lib/python3.9/site->packages/torchvision/datasets/cifar.py:118, in CIFAR10.getitem(self, index)
115 img = Image.fromarray(img)
117 if self.transform is not None:
--> 118 img = self.transform(img)
120 if self.target_transform is not None:
121 target = self.target_transform(target)
File ~/miniconda3/envs/resnet/lib/python3.9/site->packages/torchvision/transforms/transforms.py:95, in Compose.call(self, img)
93 def call(self, img):
94 for t in self.transforms:
---> 95 img = t(img)
96 return img
File ~/miniconda3/envs/resnet/lib/python3.9/site->packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks >or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []
File ~/miniconda3/envs/resnet/lib/python3.9/site->packages/torchvision/transforms/transforms.py:707, in RandomHorizontalFlip.forward(self, >img)
699 def forward(self, img):
700 """
701 Args:
702 img (PIL Image or Tensor): Image to be flipped.
(...)
705 PIL Image or Tensor: Randomly flipped image.
706 """
--> 707 if torch.rand(1) < self.p:
708 return F.hflip(img)
709 return img
TypeError: '<' not supported between instances of 'Tensor' and 'list'
I was having the same error message, probably under different circumstances, but I just found my own bug and figured I would share it anyway for various readers. I was using a torchvision transformation in my dataset, which the dataloader was loading from. The transformation was
torchvision.transforms.RandomHorizontalFlip([0.5]),
and the error is that the input to this transformation should not be a list but should be
torchvision.transforms.RandomHorizontalFlip(0.5),
So if there is anything I can recommend, it's just that maybe there is some list argument being passed through that shouldn't be in some transformation or otherwise.
I have a 2D JAX array containing an image.
For each pixel P[y, x] of the image, I would like to loop over all pixels P[y, x-i] to the left of that pixel and reduce those to a single value. The exact reduction computation involves finding a particular maximum over a weighted sum involving those pixels' values, as well as i and x. Therefore, the result (or any intermediate results) for P[y, x] can't be reused for P[y, x+1] either; this is an O(x²y) operation overall.
Can I accomplish this somewhat efficiently in JAX? If so, how?
JAX does not provide any native tool to do this sort of operation for an arbitrary function. It can be done via lax.scan or perhaps jnp.cumsum for functions where each successive value can be computed from the last, but it sounds like that is not the case here.
I believe the best you can do is to combine vmap with Python for-loops to achieve what you want: just be aware that during JIT compilation JAX will flatten all for loops, so if your image size is very large, the compilation time will be long. Here's a short example:
import jax.numpy as jnp
from jax import vmap
def reduction(x):
# some 1D reduction
assert x.ndim == 1
return len(x) + jnp.sum(x)
def cumulative_apply(row, reduction=reduction):
return jnp.array([reduction(row[:i]) for i in range(1, len(row) + 1)])
P = jnp.arange(20).reshape(4, 5)
result = vmap(cumulative_apply)(P)
print(result)
# [[ 1 3 6 10 15]
# [ 6 13 21 30 40]
# [11 23 36 50 65]
# [16 33 51 70 90]]
When i am running this script-->
tokenizer.fit_on_texts(df['text'].values)
sequences = tokenizer.texts_to_sequences(df['text'].values)
word_index = tokenizer.word_index
print('Found %s unique tokens.' % len(word_index))
I am getting this error
AttributeError Traceback (most recent call
last)
<ipython-input-4-7c08b89b116a> in <module>()
----> 1 tokenizer.fit_on_texts(df['text'].values)
2 sequences = tokenizer.texts_to_sequences(df['text'].values)
3 word_index = tokenizer.word_index
4 print('Found %s unique tokens.' % len(word_index))
/opt/conda/lib/python3.6/site-packages/keras_preprocessing/text.py in
fit_on_texts(self, texts)
220 self.filters,
221 self.lower,
--> 222 self.split)
223 for w in seq:
224 if w in self.word_counts:
/opt/conda/lib/python3.6/site-packages/keras_preprocessing/text.py in
text_to_word_sequence(text, filters, lower, split)
41 """
42 if lower:
---> 43 text = text.lower()
44
45 if sys.version_info < (3,):
AttributeError: 'float' object has no attribute 'lower'
My size of CSV file is 6970963 when I reduce the size it works, is there any size limit of keras Tokenizer or I am doing something wrong
I guess file size is not the issue, try using a try block and look at the data your are passing. Use some thing like this instead of the line
#instead of this
tokenizer.fit_on_texts(df['text'].values)
#use this to look at the data when it is causing that error.
try:
tokenizer.fit_on_texts(df['text'].values)
except Exception as e:
print("exceiption is", e)
print("data passedin ", df['text'].values)
Then you can accordingly fix the error you are getting.
Check the datatype of the text you are fitting the tokenizer on. It sees it as a float instead of string. You need to convert to string before fitting a tokenizer on it.
Try something like this:
train_x = [str(x[1]) for x in train_x]
Although it is an old thread, but still following could be answer.
You data may have nan, which are interpreted as a float instead of nan. either force the type as str(word) or remove the nan using data.fillna('empty')
I am using Theano/Pylearn2 to implement LSTM model inside my own network. However, I've found that Theano scan is much, much slower than using plain loops. I used the Theano profiler
<% time> <sum %> <apply time> <time per call> <type> <#call> <#apply> <Class name>
95.4% 95.4% 25.255s 4.31e-02s Py 586 3 theano.scan_module.scan_op.Scan
1.8% 97.2% 0.466s 4.72e-05s C 9864 41 theano.sandbox.cuda.basic_ops.GpuElemwise
0.8% 97.9% 0.199s 8.75e-05s C 2276 10 theano.sandbox.cuda.basic_ops.GpuAlloc
0.7% 98.7% 0.196s 1.14e-04s C 1724 8 theano.sandbox.cuda.blas.GpuDot22
0.3% 99.0% 0.087s 1.06e-04s C 828 3 theano.sandbox.cuda.basic_ops.GpuIncSubtensor
0.2% 99.2% 0.051s 1.66e-04s Py 310 2 theano.sandbox.cuda.basic_ops.GpuAdvancedSubtensor1
and the Ops,
<% time> <sum %> <apply time> <time per call> <type> <#call> <#apply> <Op name>
77.2% 77.2% 20.433s 7.40e-02s Py 276 1 forall_inplace,gpu,grad_of_lstm__layers}
18.2% 95.4% 4.822s 1.56e-02s Py 310 2 forall_inplace,gpu,lstm__layers}
So lots and lots of time are spent on Scan (which is kind of as expected, but I didn't expect it to be soo slow).
The main body of my code is
def fprop(self, state_below, state_prev = 0, cell_prev = 0):
if state_prev == None:
state_prev = self.state_prev;
if cell_prev == None:
cell_prev = self.cell_prev;
i_gate = T.nnet.sigmoid(T.dot(state_below,self.Wi) +
T.dot(state_prev,self.Ui));
f_gate = T.nnet.sigmoid(T.dot(state_below,self.Wf) +
T.dot(state_prev,self.Uf));
C = T.tanh(T.dot(state_below, self.Wc) +
T.dot(state_prev, self.Uc));
C = i_gate * C + f_gate * cell_prev;
o_gate = T.nnet.sigmoid(T.dot(state_below,self.Wo) +
T.dot(state_prev,self.Uo) +
T.dot(C, self.Vo));
h_out = o_gate * T.tanh(C);
return h_out, C
And I wrote my scan as:
[h,c,out], _ = theano.scan(fn=self.fprop_with_output,
sequences=[X.T,Y[:,1:].T],
outputs_info=[dict(initial=h_,taps=[-1]), dict(initial=c_,taps=[-1]),None],n_steps=X.shape[1]-1);
One thing I've noticed is that the type of Theano scan uses Python implementation (?) is that the reason why this is ridiculously slow? or did I do something wrong? Why is Theano python implementation of Scan instead of C's.
(I said using loops is faster, but it's faster at runtime, for large model I didn't manage to compile the version of using loops within reasonable amount of time).
This was asked a while ago but I had/have the same problem. Answer is that scan is slow on the GPU.
See: https://github.com/Theano/Theano/issues/1168
It takes time for Theano developers to implement scan and gradient-of-scan using C and GPU, because it is much more complicated than other functions. That is why when you profile it, it shows GpuElemwise, GpuGemv, GpuDot22, etc., but you don't see a GpuScan or GpuGradofScan.
Meanwhile, you can only fall back to for loops.