Categories
Misc

Image data generator

Hey, so as to my understanding of how IamgeDataGenerator works , is that it choose a random operation and apply it to the image. With this also applies to rescale, std_normalistion and zac_whiteing, with those not be applied to all the images?

submitted by /u/Ali99695
[visit reddit] [comments]

Categories
Misc

Update : 300 Backers! Announcing 4 Stretch Goals! · Deep Learning With TensorFlow & Keras

Update : 300 Backers! Announcing 4 Stretch Goals! · Deep Learning With TensorFlow & Keras submitted by /u/spmallick
[visit reddit] [comments]
Categories
Misc

Custom loss function for tensorflow ML

I am a novice to Tensorflow but have significant experience in the scikit-learn environment. I am trying to shift some codes to the more flexible TF/Keras routines in Python. One modification I would like to do is to build a simple neural network that trains on an earth mover’s distance/Wasserstein distance between arrays representing densities, rather than on direct root-mean-square differences in the densities.

I have been able to modify a loss function, but do not know how to implement the EMD in TF. I use pyEMD to calculate the EMD between two of the arrays in Python, but is doesn’t translate to tensors in TF. It seems like I may have to “unstack” the tensors, evaluate by each row, then “restack” as a tensor. It doesn’t seem reasonable to do it this way, but I am willing to make it work. At this point, I am focused on making it work rather than be as-fast-as-possible.

Does anyone have advice on either a TF-friendly EMD approach or how I may use pyEMD in a custom loss function? I appreciate any and all suggestions.

submitted by /u/baudie
[visit reddit] [comments]

Categories
Misc

Guinness World Record Awarded for Fastest DNA Sequencing — Just 5 Hours

Guinness World Records this week presented a Stanford University-led research team with the first record for fastest DNA sequencing technique — a benchmark set using a workflow sped up by AI and accelerated computing. Achieved in five hours and two minutes, the DNA sequencing record can allow clinicians to take a blood draw from a Read article >

The post Guinness World Record Awarded for Fastest DNA Sequencing — Just 5 Hours appeared first on The Official NVIDIA Blog.

Categories
Offsites

Robust Routing Using Electrical Flows

In the world of networks, there are models that can explain observations across a diverse collection of applications. These include simple tasks such as computing the shortest path, which has obvious applications to routing networks but also applies in biology, e.g., where the slime mold Physarum is able to find shortest paths in mazes. Another example is Braess’s paradox — the observation that adding resources to a network can have an effect opposite to the one expected — which manifests not only in road networks but also in mechanical and electrical systems. For instance, constructing a new road can increase traffic congestion or adding a new link in an electrical circuit can increase voltage. Such connections between electrical circuits and other types of networks have been exploited for various tasks, such as partitioning networks, and routing flows.

In “Robust Routing Using Electrical Flows”, which won the Best Paper Award at SIGSPATIAL 2021, we present another interesting application of electrical flows in the context of road network routing. Specifically, we utilize ideas from electrical flows for the problem of constructing multiple alternate routes between a given source and destination. Alternate routes are important for many use cases, including finding routes that best match user preferences and for robust routing, e.g., routing that guarantees finding a good path in the presence of traffic jams. Along the way, we also describe how to quickly model electrical flows on road networks.

Existing Approaches to Alternate Routing
Computing alternate routes on road networks is a relatively new area of research and most techniques rely on one of two main templates: the penalty method and the plateau method. In the former, alternate routes are iteratively computed by running a shortest path algorithm and then, in subsequent runs, adding a penalty to those segments already included in the shortest paths that have been computed, to encourage further exploration. In the latter, two shortest path trees are built simultaneously, one starting from the origin and one from the destination, which are used to identify sequences of road segments that are common to both trees. Each such common sequence (which are expected to be important arterial streets for example) is then treated as a visit point on the way from the origin to the destination, thus potentially producing an alternate route. The penalty method is known to produce results of high quality (i.e., average travel time, diversity and robustness of the returned set of alternate routes) but is very slow in practice, whereas the plateau method is much faster but results in lower quality solutions.

An Alternate to Alternate Routing: Electrical Flows
Our approach is different and assumes that a routing problem on a road network is in many ways analogous to the flow of electrical current through a resistor network. Though the electrical current travels through many different paths, it is weaker along paths of higher resistance and stronger on low resistance ones, all else being equal.

We view the road network as a graph, where intersections are nodes and roads are edges. Our method then models the graph as an electrical circuit by replacing the edges with resistors, whose resistances equal the road traversal time, and then connecting a battery to the origin and destination, which results in electrical current between those two points. In this analogy, the resistance models how time-consuming it is to traverse a segment. In this sense, long and congested segments have high resistances. Intuitively speaking, the flow of electrical current will be spread around the entire network but concentrated on the routes that have lower resistance, which correspond to faster routes. By identifying the primary routes taken by the current, we can construct a viable set of alternates from origin to destination.

Example of how we construct the electrical circuit corresponding to the road network. The current can be decomposed into three flows, i1, i2 and i3; each of which corresponds to a viable alternate path from Fremont to San Rafael.

In order to compute the electrical flow, we use Kirchhoff’s and Ohm’s laws, which say respectively: 1) the algebraic sum of currents at each junction is equal to zero, meaning that the traffic that enters any intersection also exits it (for instance if three cars enter an intersection from one street and another car enters the same intersection from another street, a total of four cars need to exit the intersection); and 2) the current is directly proportional to the voltage difference between endpoints. If we write down the resulting equations, we end up with a linear system with n equations over n variables, which correspond to the potentials (i.e, the voltage) at each intersection. While voltage has no direct analogy to road networks, it can be used to help compute the flow of electrical current and thus find alternate routes as described above.

In order to find the electrical current i (or flow) on each wire, we can use Kirchhoff’s law and Ohm’s law to obtain a linear system of equations in terms of voltages (or potentials) v. This yields a linear system with three equations (representing Kirchhoff’s law) and three unknowns (voltages at each intersection).

So the computation boils down to computing values for the variables of this linear system involving a very special matrix called Laplacian matrix. Such matrices have many useful properties, e.g., they are symmetric and sparse — the number of off-diagonal non-zero entries is equal to twice the number of edges. Even though there are many existing near-linear time solvers for such systems of linear equations, they are still too slow for the purposes of quickly responding to routing requests with low latency. Thus we devised a new algorithm that solves these linear systems much faster for the special case of road networks1.

Fast Electrical Flow Computation
The first key part of this new algorithm involves Gaussian elimination, which is possibly the most well-known method for solving linear systems. When performed on a Laplacian matrix corresponding to some resistor network, it corresponds to the Y-Δ transformation, which reduces the number of nodes, while preserving the voltages. The only downside is that the number of edges may increase, which would make the linear system even slower to solve. For example, if a node with 10 connections is eliminated using the Y-Δ transformation, the system would end up with 35 new connections!

The Y-Δ transformation allows us to remove the middle junction and replace it with three connections (Ra, Rb and Rc) between N1, N2 and N3. (Image from Wikipedia)

However if one can identify parts of the network that are connected to the rest through very few nodes (lets call these connections bottlenecks), and perform elimination on everything else while leaving the bottleneck nodes, the new edges formed at the end will only be between bottleneck nodes. Provided that the number of bottleneck nodes is much smaller than the number of nodes eliminated with Y-Δ — which is true in the case of road networks since bottleneck nodes, such as bridges and tunnels, are much less common than regular intersections — this will result in a large net decrease (e.g., ~100x) in terms of graph size. Fortunately, identifying such bottlenecks in road networks can be done easily by partitioning such a network. By applying Y-Δ transformation to all nodes except the bottlenecks2, the result is a much smaller graph for which the voltages can be solved faster.

But what about computing the currents on the rest of the network, which is not made up of bottleneck nodes? A useful property about electrical flows is that once the voltages on bottleneck nodes are known, one can easily compute the electrical flow for the rest of the network. The electrical flow inside a part of the network only depends on the voltage of bottleneck nodes that separate that part from the rest of the network. In fact, it’s possible to precompute a small matrix so that one can recover the electrical flow by a single matrix-vector multiplication, which is a very fast operation that can be run in parallel.

Consider the imposed conceptual road network on Staten Island (left), for which directly computing the electrical flow would be slow. The bridges (red nodes) are the bottleneck points, and we can eliminate the whole road network inside the island by repeatedly applying Gaussian Elimination (or Y-Δ transformation). The resulting network (middle) is a much smaller graph, which allows for faster computation. The potentials inside the eliminated part are always a fixed linear combination of the bottleneck nodes (right).

Once we obtain a solution that gives the electrical flow in our model network, we can observe the routes that carry the highest amount of electrical flow and output those as alternate routes for the road network.

Results
Here are some results depicting the alternates computed by the above algorithm.

Different alternates found for the Bay Area. Different colors correspond to different routes from the origin (red icon toward the bottom) to the destination (blue icon toward the top).

Conclusion
In this post we describe a novel approach for computing alternate routes in road networks. Our approach is fundamentally different from the main techniques applied in decades of research in the area and provides high quality alternate routes in road networks by studying the problem through the lens of electrical circuits. This is an approach that can prove very useful in practical systems and we hope inspires more research in the area of alternate route computation and related problems. Interested readers can find a more detailed discussion of this work in our SIGSPATIAL 2021 talk recording.

Acknowledgements
We thank our collaborators Lisa Fawcett, Sreenivas Gollapudi, Ravi Kumar, Andrew Tomkins and Ameya Velingker from Google Research.


1Our techniques work for any network that can be broken down to smaller components with the removal of a few nodes. 
2 Performing Y-Δ transformation one-by-one for each node will be too slow. Instead we eliminate whole groups of nodes by taking advantage of the algebraic properties of Y-Δ transformation. 

Categories
Misc

What happens when number of neurons in the last layer of the network don’t match the length of labels?

For example I can have the last layer of my network be tf.keras.layers.Dense(10) even if my labels are just single values, so the y is shape (500,)

The last layer outputs 10 values but there is only one label per input vector.

How can this work? Which of the 10 values is compared to the label to calculate the loss? I expected keras to give me an error in this case but it trains just fine and even does well on the toy dataset I’m using.

submitted by /u/woodenlathe
[visit reddit] [comments]

Categories
Misc

Take the Future for a Spin at GTC 2022

Don’t miss the chance to experience the breakthroughs that are driving the future of autonomy. NVIDIA GTC will bring together the leaders, researchers and developers who are ushering in the era of autonomous vehicles. The virtual conference, running March 21-24, also features experts from industries transformed by AI, such as healthcare, robotics and finance. And Read article >

The post Take the Future for a Spin at GTC 2022 appeared first on The Official NVIDIA Blog.

Categories
Misc

Tensorflow GPU failing with different error flavors but CPU works

So I have been trying at this for ages now, my GPU is GTX 1650 SUPER and I have Python 3.9.5 with tensorflow version 2.7.0 and CUDA 11.2, cuDNN 8.1.0. Here is the issue: sometimes, my models train but most of the time it errors with:

E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 

or some other variants (again, they kinda take turns fighting to be errors):

E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_INSTRUCTION: an illegal instruction was encountered 

and I have absolutely no idea why its doing this. I don’t think its my code because I was able to train my code 100% of the time on CPU, but it takes like half an hour to do that and GPUs take only 3 minutes, 10 times faster!!! Who else has this error and why is it occuring randomly, also: could MSI Afterburner mess with it?

submitted by /u/bootsareme
[visit reddit] [comments]

Categories
Misc

Trouble Importing JSON File of Annotated Images

I have a small data set of images (seven images) that I annotated by drawing bounding boxes on the objects I want to detect using VGG Image Annotator (VIA). When I exported the annotations, it only gave me one JSON file for all the annotations.

How would I go about importing that JSON file, along with the pictures into TensorFlow? I tried to follow the documentation, but got lost.

I also tried the RoboFlow service where I could turn my images/annotations into TFRecords and a TensorFlow CSV files, but I was also lost on how to implement this.

I am using Python 3.6.9 in my JupyterLab environment.

submitted by /u/Tuka-Cola
[visit reddit] [comments]

Categories
Misc

React Native mobile app with a Python backend running the model?

I trained a custom image classification model using Python and now want to deploy it on a native application which I’ve made using React Native. My initial approach to this was to use tfjs. However, tfjs has poor support, unresolved complaints on the forums and a ton of dependency issues when it comes to using custom models on react native, from my experience. I couldn’t figure out any of those by myself. So I switched up and am now trying to load the model on a Python REST API, and am planning to call it from my React Native app. Is this the best way to approach this problem? I’m new to machine learning and programming overall in general, and would love to hear some insight. Is it sensible to upload an image from the React Native front end and then get the prediction using the model on the back end? Also if I’m wrong about the problems with tfjs and someone has successfully used it I’d love to hear how you do it, please. Any and everything helpful is much appreciated, thank you very much.

submitted by /u/NoobyTooo
[visit reddit] [comments]