sounddevice.PortAudioError: Error opening OutputStream: Internal PortAudio error [PaErrorCode -9986] - portaudio

I am on MacMojave and try to play a sound file if a microphone signal goes above a threshold. I have :
import sounddevice as sd
import soundfile as sf
import numpy as np
duration = 5 # seconds
threshold = 30
filename = "./data/piano.wav"
data, fs = sf.read(filename, dtype='float32')
def print_sound(indata, outdata, frames, time, status):
if status:
print(status)
global threshold, data, fs
volume_norm = 5*np.linalg.norm(indata)
if volume_norm > threshold:
print("so loud!!!!")
sd.play(data, fs)
with sd.Stream(callback=print_sound):
sd.sleep(1000*duration)
When I run this the sound goes above threshold, however, it cannot open the output Stream as follows;
$python3 main.py
input underflow
input underflow
input underflow
so loud!!!!
so loud!!!!
input overflow, output underflow
so loud!!!!
input overflow, output underflow
so loud!!!!
||PaMacCore (AUHAL)|| Error on line 1316: err=''nope'', msg=Audio Hardware: Illegal Operation
From cffi callback <function _StreamBase.__init__.<locals>.callback_ptr at 0x10caf8680>:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sounddevice.py", line 881, in callback_ptr
callback, idata, odata, frames, time, status)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sounddevice.py", line 2678, in _wrap_callback
callback(*args)
File "main.py", line 17, in print_sound
sd.play(data, fs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sounddevice.py", line 177, in play
**kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sounddevice.py", line 2578, in start_stream
**kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sounddevice.py", line 1489, in __init__
**_remove_self(locals()))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sounddevice.py", line 895, in __init__
'Error opening {}'.format(self.__class__.__name__))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sounddevice.py", line 2738, in _check
raise PortAudioError(errormsg, err)
sounddevice.PortAudioError: Error opening OutputStream: Internal PortAudio error [PaErrorCode -9986]
I have restarted my iMac ( Mojave) and still get the same error.
Any help would be much appreciated.
CS

Related

How to create a dask-array from CuPy array?

I'm trying to launch dask.cluster.Kmeans with the huge amount of data.
Working with CPU is OK since i wrap numpy arrays with dask.array.
Working with GPU doesn't seem to be possible due to not implemented functionalities in cupy.
I've tried to reproduce Mattew Rocklin example (https://blog.dask.org/2019/01/03/dask-array-gpus-first-steps) on generating random dask array from CuPy random generator - and it works, but it's not the case I want to use.
Wrapping cupy with dask.array - doesn't work.
>>> import dask.array as da
>>> import cupy as cp
>>> da.from_array(cp.arange(100000)).sum().compute()
I expect the sum of this array but get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/base.py", line 175, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/base.py", line 446, in compute
results = schedule(dsk, keys, **kwargs)
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/threaded.py", line 82, in get
**kwargs
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/local.py", line 491, in get_async
raise_exception(exc, tb)
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/compatibility.py", line 130, in reraise
raise exc
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/local.py", line 233, in execute_task
result = _execute_task(task, data)
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/core.py", line 119, in _execute_task
return func(*args2)
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/dask/array/core.py", line 100, in getter
c = np.asarray(c)
File "/home/ubuntu/miniconda3/envs/cupy/lib/python3.6/site-packages/numpy/core/numeric.py", line 538, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: object __array__ method not producing an array
So how could I manage the work with CuPy through the dask array?
When creating the Dask Array from a CuPy array, you need to supply da.from_array the keyword argument asarray=False. So your code would look like the following.
>>> import dask.array as da
>>> import cupy as cp
>>> da.from_array(cp.arange(100000), asarray=False).sum().compute()

Dask - Drop duplicate index MemoryError

I'm getting a MemoryError when I try to drop duplicate timestamps on a large dataframe with the following code.
import dask.dataframe as dd
path = f's3://{container_name}/*'
ddf = dd.read_parquet(path, storage_options=opts, engine='fastparquet')
ddf = ddf.reset_index().drop_duplicates(subset='timestamp_utc').set_index('timestamp_utc')
...
Profiling shows that it was using up about 14GB of RAM on a dataset of 265MB of gzipped parquet files containing about 40 million rows of data.
Is there an alternative way I can drop duplicate indexes on my data without Dask using so much memory?
The traceback below
Traceback (most recent call last):
File "/anaconda/envs/surb/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/anaconda/envs/surb/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/chengkai/surbana_lift/src/consolidate_data.py", line 62, in <module>
consolidate_data()
File "/home/chengkai/surbana_lift/src/consolidate_data.py", line 37, in consolidate_data
ddf = ddf.reset_index().drop_duplicates(subset='timestamp_utc').set_index('timestamp_utc')
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/dataframe/core.py", line 2524, in set_index
divisions=divisions, **kwargs)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/dataframe/shuffle.py", line 64, in set_index
divisions, sizes, mins, maxes = base.compute(divisions, sizes, mins, maxes)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/base.py", line 407, in compute
results = get(dsk, keys, **kwargs)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/threaded.py", line 75, in get
pack_exception=pack_exception, **kwargs)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/local.py", line 521, in get_async
raise_exception(exc, tb)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/compatibility.py", line 67, in reraise
raise exc
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/local.py", line 290, in execute_task
result = _execute_task(task, data)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/local.py", line 270, in _execute_task
args2 = [_execute_task(a, cache) for a in args]
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/local.py", line 270, in <listcomp>
args2 = [_execute_task(a, cache) for a in args]
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/local.py", line 267, in _execute_task
return [_execute_task(a, cache) for a in arg]
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/local.py", line 267, in <listcomp>
return [_execute_task(a, cache) for a in arg]
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/local.py", line 271, in _execute_task
return func(*args2)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/dataframe/core.py", line 69, in _concat
return args[0] if not args2 else methods.concat(args2, uniform=True)
File "/anaconda/envs/surb/lib/python3.6/site-packages/dask/dataframe/methods.py", line 329, in concat
out = pd.concat(dfs3, join=join)
File "/anaconda/envs/surb/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 226, in concat
return op.get_result()
File "/anaconda/envs/surb/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 423, in get_result
copy=self.copy)
File "/anaconda/envs/surb/lib/python3.6/site-packages/pandas/core/internals.py", line 5418, in concatenate_block_manage
rs
[ju.block for ju in join_units], placement=placement)
File "/anaconda/envs/surb/lib/python3.6/site-packages/pandas/core/internals.py", line 2984, in concat_same_type
axis=self.ndim - 1)
File "/anaconda/envs/surb/lib/python3.6/site-packages/pandas/core/dtypes/concat.py", line 461, in _concat_datetime
return _concat_datetimetz(to_concat)
File "/anaconda/envs/surb/lib/python3.6/site-packages/pandas/core/dtypes/concat.py", line 506, in _concat_datetimetz
new_values = np.concatenate([x.asi8 for x in to_concat])
MemoryError
It is not too surprising that the data becomes very big in memory. Parquet is a pretty efficient format in terms of space, especially with gzip compression, and strings all become python objects (so expensive in memory).
In addition, you have a number of worker threads operating on parts of the overall dataframe. That involves data copying, intermediates, and concatenation of results; the latter being pretty inefficient in pandas.
One suggestion: instead of reset_index, you can remove one step by specifying index=False to read_parquet.
Next suggestion: limit the number of threads you use to a smaller number than the default, which is probably your number of CPU cores. The easiest way to do that is to use the distributed client in-process
from dask.distributed import Client
c = Client(processes=False, threads_per_worker=4)
It may be better to set the index first, and then do the drop_duplicated with map_partitions to minimise cross-partition communication.
df.map_partitions(lambda d: d.drop_duplicates(subset='timestamp_utc'))

dask dataframe set_index throws error

I have a dask dataframe created from parquet file on HDFS.
When creating setting index using api: set_index, it fails with below error.
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/dask/dataframe/shuffle.py", line 64, in set_index
divisions, sizes, mins, maxes = base.compute(divisions, sizes, mins, maxes)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/dask/base.py", line 206, in compute
results = get(dsk, keys, **kwargs)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/distributed/client.py", line 1949, in get
results = self.gather(packed, asynchronous=asynchronous)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/distributed/client.py", line 1391, in gather
asynchronous=asynchronous)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/distributed/client.py", line 561, in sync
return sync(self.loop, func, *args, **kwargs)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/distributed/utils.py", line 241, in sync
six.reraise(*error[0])
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/distributed/utils.py", line 229, in f
result[0] = yield make_coro()
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/distributed/client.py", line 1269, in _gather
traceback)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/ebs/d1/agent/conda/envs/py361/lib/python3.6/site-packages/dask/dataframe/io/parquet.py", line 144, in _read_parquet_row_group
open=open, assign=views, scheme=scheme)
TypeError: read_row_group_file() got an unexpected keyword argument 'scheme'
Can some one point me to the reason of this error and how to fix it.
Solution
Upgrade fastparquet to version 0.1.3.
Details
Dask 0.15.4, used for your example, includes this commit, which adds the argument scheme to read_row_group_file(). This throws an error for fastparquet versions before 0.1.3.

TypeError: Value passed to parameter 'input' has DataType string not in list of allowed values: int32, int64, complex64, float32, float64, bool, int8

I was trying to use tensorflow. The input attributes are similar to census example except that the LABEL Column is a continuous value. I executed the below command:
test-server#:~/aaaml-samples/arbitrator$ gcloud ml-engine local train --module-name trainer.task --package-path trainer/ -- --train-files $TRAIN_DATA --eval-files $EVAL_DATA --train-steps 1000 --job-dir
$MODEL_DIR
Filename: ['/home/madhukar_mhraju/aaaml-samples/arbitrator/data/aaa.data.csv']
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Filename: ['/home/madhukar_mhraju/aaaml-samples/arbitrator/data/aaa.test.csv']
Filename: ['/home/madhukar_mhraju/aaaml-samples/arbitrator/data/aaa.test.csv']
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/madhukar_mhraju/aaaml-samples/arbitrator/trainer/task.py", line 193, in <module>
learn_runner.run(generate_experiment_fn(**arguments), job_dir)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 106, in run
return task()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 465, in train_and_evaluate
export_results = self._maybe_export(eval_result)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 484, in _maybe_export
compat.as_bytes(strategy.name))))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/export_strategy.py", line 32, in export
return self.export_fn(estimator, export_path)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/utils/saved_model_export_utils.py", line 283, in export_fn
exports_to_keep=exports_to_keep)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/framework/experimental.py", line 64, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1264, in export_savedmodel
model_fn_lib.ModeKeys.INFER)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1133, in _call_model_fn
model_fn_results = self._model_fn(features, labels, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 268, in _dnn_linear_combined_model_fn
scope=scope)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/feature_column_ops.py", line 531, in weighted_sum_from_feature_columns
transformed_tensor = transformer.transform(column)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/feature_column_ops.py", line 879, in transform
feature_column.insert_transformed_feature(self._columns_to_tensors)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/feature_column.py", line 528, in insert_transformed_feature
sparse_values = string_ops.as_string(input_tensor.values)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_string_ops.py", line 51, in as_string
width=width, fill=fill, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 585, in apply_op
param_name=input_name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 61, in _SatisfiesTypeConstraint
", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: Value passed to parameter 'input' has DataType string not
in list of allowed values: int32, int64, complex64, float32, float64,
bool, int8
Am new to tensorflow. I understand that this issue is occurring while processing the evaluation file(aaa.test.csv). The evaluation file data and format is correctly defined. And also the column data type have been mapped correctly as well.But i am not sure why the error is occurring.
1) The training data csv had column headings in them. When I generated the data, i was reordering them randomly, which resulted in the column headings being moved to somewhere in the middle. Hence the type error. It was difficult to find out as the training data was huge.

Tensorflow: ValueError when using AdamOptimizer

When running a session in TensorFlow I get the following error
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 190, in minimize
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 241, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/ops/gradients.py", line 481, in gradients
in_grads = _AsList(grad_fn(op, *out_grads))
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/ops/array_grad.py", line 162, in _DiagGrad
return array_ops.diag_part(grad)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 380, in diag_part
return _op_def_lib.apply_op("DiagPart", input=input, name=name)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
op_def=op_def)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2156, in create_op
set_shapes_for_outputs(ret)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1612, in set_shapes_for_outputs
shapes = shape_func(op)
File "/local0/software/python/python_bleeding_edge/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 982, in _DiagPartShape
" do not match ")
ValueError: Invalid shape, shape[:mid] (?,) and shape[mid:] (?,) do not match
I am not sure where it comes from, since it does not give any error indication in the model construction. I've also tried different optimisers, e.g. GradientDescentOptimizer but the error persists.
Actually error is somehow obvious as said in the error
_DiagPartShape Invalid shape, shape[:mid] (?,) and shape[mid:] (?,) do not match
You provided wrong dimensions.

Resources