Categories
Misc

Make a Digital Twin of your Data Center with SONiC running on NVIDIA Air

We have made it incredibly easy to try-out a full multi-switch network fabric using the Microsoft SONiC operating system – in a virtual data center that is available to anyone free of charge.

Testing out new network gear and running proof of concept (POC) tests for new technology can be difficult at the best of times, but in today’s environment, it’s even harder. We have made it incredibly easy to try-out a full multi-switch network fabric using the Microsoft SONiC operating system – in a virtual data center that is available to anyone free of charge.

Data centers serve a crucial role in business growth and organizations aim to adopt the emerging open networking mindset to enable flexibility to suit their unique business needs and operations.

For an open networking data center to occur, IT departments need to train their staff to plan their networking core replacement and predict future challenges in a relatively short time.

NVIDIA supports “Pure SONiC”, a fully open-source version of SONiC. The Pure SONiC NOS would be ideal for IT departments who don’t want to have another proprietary NOS and want to have full control and flexibility in their data center. Being one of the most significant contributors to “SONiC”, NVIDIA has launched the “SONiC Air” platform to support organizations before a transition using a digital twin and provide an entire network experience. 

What’s Supported?

  • Full CLI and API functionality
  • Control plane software including BGP, VLANs and containers
  • Automation and Zero Touch Provisioning (ZTP)
  • Network monitoring with streaming telemetry
  • Interop testing between NVIDIA Cumulus Linux and SONiC
  • Custom topologies and network designs

Using “SONiC Air” enables flexibility in the evaluation process, eliminating the limitations of having a small POC that is not representative of the production environment. Staff can use the platform for free, build an exact network digital twin, validate configurations, confirm security policies or test CI/CD pipelines. In addition to CLI access, the platform provides full software functionality and access to the system core components such as docker containers and APIs. On the other hand, since the platform is software-based it doesn’t support hardware features like “What Just Happened”, to let the end-user know why the ASIC has dropped a packet and assist in troubleshooting.

Beyond POCs, with NVIDIA Air, customers can build a tailor-made network topology, define any connectivity, and create configurations and automation for the initial deployment and ongoing operations, before any hardware even ships. Today’s customers are building their entire network with a digital twin and enabling services the same day equipment is installed.

Planning

Organizations who choose to define their workflow pipelines in advance can dramatically reduce the transition time to an open networking product deployed and fully operational in their production environment and save on both CapEx and OpEx in the long term.

Organizations can use NVIDIA Air for end-to-end evaluation and testing, combining the network,servers and applications. NVIDIA Air not only supports SONiC and Cumulus Linux on the network but also Ubuntu and Red Hat servers. Leveraging Infrastructure As Code, the platform supports integration with production’s CI/CD pipeline and version control repository to test the integrations and build the code for their future production environment. 

Staff training 

Training resources are always a challenge for IT departments. With the use of NVIDIA Air, IT teams can now give every team member their own private replica of the production environment to learn on. No more waiting for hardware resources to be racked and stacked or balancing limited lab time across multiple users

Get started

Customers who want to try SONiC in NVIDIA Air can watch the “SONiC experience” on-demand, hands-on workshop to help them get started. The workshop covers the basics of SONiC architecture, configuration and troubleshooting. We demystify the SONiC microservices architecture and highlight the different configuration approaches available in SONiC. The hands-on lab provides a step-by-step guide to build and configure a leaf-spine SONiC network from the ground up.

Links

Categories
Misc

NVIDIA Announces Financial Results for First Quarter Fiscal 2022

NVIDIA today reported record revenue for the first quarter ended May 2, 2021, of $5.66 billion, up 84 percent from a year earlier and up 13 percent from the previous quarter, with record revenue from the company’s Gaming, Data Center and Professional Visualization platforms.

Categories
Offsites

Cross-Modal Contrastive Learning for Text-to-Image Generation

Posted by Han Zhang, Research Scientist and Jing Yu Koh, Software Engineer, Google Research

Automatic text-to-image synthesis, in which a model is trained to generate images from text descriptions alone, is a challenging task that has recently received significant attention. Its study provides rich insights into how machine learning (ML) models capture visual attributes and relate them to text. Compared to other kinds of inputs to guide image creation, such as sketches, object masks or mouse traces (which we have highlighted in prior work), descriptive sentences are a more intuitive and flexible way to express visual concepts. Hence, a strong automatic text-to-image generation system can also be a useful tool for rapid content creation and could be applied to many other creative applications, similar to other efforts to integrate machine learning into the creation of art (e.g., Magenta).

State-of-the-art image synthesis results are typically achieved using generative adversarial networks (GANs), which train two models — a generator, which tries to create realistic images, and a discriminator, which tries to determine if an image is real or fabricated. Many text-to-image generation models are GANs that are conditioned using text inputs in order to generate semantically relevant images. This is significantly challenging, especially when long, ambiguous descriptions are provided. Moreover, GAN training can be prone to mode collapse, a common failure case for the training process in which the generator learns to produce only a limited set of outputs, so that the discriminator fails to learn robust strategies to recognize fabricated images. To mitigate mode collapse, some approaches use multi-stage refinement networks that iteratively refine an image. However, such systems require multi-stage training, which is less efficient than simpler single-stage end-to-end models. Other efforts rely on hierarchical approaches that first model object layouts before finally synthesizing a realistic image. This requires the use of labeled segmentation data, which can be difficult to obtain.

In “Cross-Modal Contrastive Learning for Text-to-Image Generation,” to appear at CVPR 2021, we present the Cross-Modal Contrastive Generative Adversarial Network (XMC-GAN), which addresses text-to-image generation by learning to maximize the mutual information between image and text using inter-modal (image-text) and intra-modal (image-image) contrastive losses. This approach helps the discriminator to learn more robust and discriminative features, so XMC-GAN is less prone to mode collapse even with one-stage training. Importantly, XMC-GAN achieves state-of-the-art performance with a simple one-stage generation, as compared to previous multi-stage or hierarchical approaches. It is end-to-end trainable, and only requires image-text pairs (as opposed to labeled segmentation or bounding box data).

Contrastive Losses for Text-to-Image Synthesis
The goal of text-to-image synthesis systems is to produce clear, photo-realistic scenes with high semantic fidelity to their conditioned text descriptions. To achieve this, we propose to maximize the mutual information between the corresponding pairs: (1) images (real or generated) with a sentence describing the scene; (2) a generated image and a real image with the same description; and (3) regions of an image (real or generated) and words or phrases associated with them.

In XMC-GAN, this is enforced using contrastive losses. Similar to other GANs, XMC-GAN contains a generator for synthesizing images, and a discriminator that is trained to act as a critic between real and generated images. Three sets of data contribute to the contrastive loss in this system — the real images, the text that describes those images, and the images generated from the text descriptions. The individual loss functions for both the generator and the discriminator are combinations of the loss calculated from whole images with the full text description, combined with the loss calculated from sub-divided images with associated words or phrases. Then, for each batch of training data, we calculate the cosine similarity score between each text description and the real images, and likewise, between each text description and the batch of generated images. The goal is for the matching pairs (both text-to-image and real image-to-generated image) to have high similarity scores and for non-matching pairs to have low scores. Enforcing such a contrastive loss allows the discriminator to learn more robust and discriminative features.

Inter-modal and intra-modal contrastive learning in our proposed XMC-GAN text-to-image synthesis model.

Results
We apply XMC-GAN to three challenging datasets — the first was a collection of MS-COCO descriptions of MS-COCO images, and the other two were datasets annotated with Localized Narratives, one of which covers MS-COCO images (which we call LN-COCO) and the other of which describes Open Images data (LN-OpenImages). We find that XMC-GAN achieves a new state of the art on each. The images generated by XMC-GAN depict scenes that are of higher quality than those generated using other techniques. On MS-COCO, XMC-GAN improves the state-of-the-art Fréchet inception distance (FID) score from 24.7 to 9.3, and is significantly preferred by human evaluators.

Selected qualitative results for generated images on MS-COCO.

Similarly, human raters prefer the image quality in XMC-GAN generated images 77.3% of the time, and 74.1% prefer its image-text alignment compared to three other state-of-the-art approaches (CP-GAN, SD-GAN, and OP-GAN) .

Human evaluation on MS-COCO for image quality and text alignment. Annotators rank (anonymized and order-randomized) generated images from best to worst.

XMC-GAN also generalizes well to the challenging Localized Narratives dataset, which contains longer and more detailed descriptions. Our prior work TReCS tackles text-to-image generation for Localized Narratives using mouse trace inputs to improve image generation quality. Despite not receiving mouse trace annotations, XMC-GAN is able to significantly outperform TReCS on image generation on LN-COCO, improving state-of-the-art FID from 48.7 to 14.1. Incorporating mouse traces and other additional inputs into an end-to-end model such as XMC-GAN would be interesting to study in future work.

In addition, we also train and evaluate on the LN-OpenImages, which is more challenging than MS-COCO because the dataset is much larger with images that cover a broader range of subject matter and that are more complex (8.4 objects on average). To the best of our knowledge, XMC-GAN is the first text-to-image synthesis model that is trained and evaluated on Open Images. XMC-GAN is able to generate high quality results, and sets a strong benchmark FID score of 26.9 on this very challenging task.

Random samples of real and generated images on Open Images.

Conclusion and Future Work
In this work, we present a cross-modal contrastive learning framework to train GAN models for text-to-image synthesis. We investigate several cross-modal contrastive losses that enforce correspondence between image and text. For both human evaluations and quantitative metrics, XMC-GAN establishes a marked improvement over previous models on multiple datasets. It generates high quality images that match their input descriptions well, including for long, detailed narratives, and does so while being a simpler, end-to-end model. We believe that this represents a significant advance towards creative applications for image generation from natural language descriptions. As we continue this research, we are continually evaluating responsible approaches, potential applications and risk mitigation, in accordance with our AI Principles.

Acknowledgements
This is a joint work with Jason Baldridge, Honglak Lee, and Yinfei Yang. We would like to thank Kevin Murphy, Zizhao Zhang, Dilip Krishnan for their helpful feedback. We also want to thank the Google Data Compute team for their work on conducting human evaluations. We are also grateful for general support from the Google Research team.

Categories
Misc

Cyber Security Analysis – Beginner’s Guide to Processing Security Logs in Python

This is the last installment of the series of articles on the RAPIDS ecosystem with this being the ninth installment. The series explores and discusses various aspects of RAPIDS that allow its users solve ETL (Extract, Transform, Load) problems, build ML (Machine Learning) and DL (Deep Learning) models, explore expansive graphs, process signal and system … Continued

This is the last installment of the series of articles on the RAPIDS ecosystem with this being the ninth installment. The series explores and discusses various aspects of RAPIDS that allow its users solve ETL (Extract, Transform, Load) problems, build ML (Machine Learning) and DL (Deep Learning) models, explore expansive graphs, process signal and system log, or use SQL language via BlazingSQL to process data.

Today’s interconnected world makes us more vulnerable to cyber attacks: ever-present IoT devices record and listen to what we do, spam and phishing emails threaten us every day, and attacks on networks that steal data can lead to serious consequences. These systems produce terabytes of logs full of information that can help detect and protect vulnerable systems. Estimating on the conservative side, a medium-sized company with hundreds to thousands of interconnected devices can produce upwards of 100GB of log files per day. Also, the rate of events that get logged can reach levels counted in tens of thousands per second.

CLX (pronounced clicks) is part of the RAPIDS ecosystem that accelerates the processing and analysis of cyber logs. As part of RAPIDS, it builds on top of RAPIDS DataFrames cuDF, and further extends the capabilities of the RAPIDS ML library cuML by tapping into the latest advances in natural language processing field to organize unstructured data and build classification models.

 The previous posts in the series showcased other areas:

In this post, we introduce CLX. To help with getting familiar with CLX, we also published a cheat sheet that can be downloaded here CLX-cheatsheet, and an interactive notebook with all the current functionality of CLX showcased here.

Cybersecurity

With the advent of personalized computers, the adversarial games shifted from pure reconnaissance missions and traditional warfare to interrupting the computer systems of one’s enemy. The organizations like the National Security Agency (NSA) are full of scientists of various backgrounds that daily try to keep our national networks safe so an adversary cannot access the power grid or our banking system. At this level stakes are exceedingly high as are the defense mechanisms to prevent such attacks.

Personal computers or business networks are a different measure: while an attack on a single computer or a network might not cripple or otherwise threaten the well-being of citizens, it can have profound effects on one person’s or a business’s finances and/or future opportunities.

Many of these attacks leave a trace, a breadcrumbs trail of information that can help a business detect an attack so it can defend itself against it. After all, any attack that compromises the ability of a company to conduct a business as usual leads to lost productivity. Worse yet, if an attacker gains access to and steals intellectual property, it may cripple or completely ruin such a business.

However, how can a business that generates upwards of 100GB per day of logs keep up with all this data flood?

cyBERT

Historically, the approach was to parse the logs using Regex. And while we’re big fans of Regex per se, such an approach becomes impractical if a business needs to maintain thousands of different patterns for every single type of a log such a business collects. And that’s where it all begins: in order to detect attacks on our network we need data, and in order to get data we need to parse logs. Without data we cannot train any machine learning model. No other way around it.

BERT (or Bi-directional Encoder Representations from Transformers) model is a deep neural network introduced by Google to build a better understanding of natural language. Unlike previous approaches for solving problems in the NLP field that relied on recurrent network architectures (like LSTM – Long Short-Term Memory), the BERT model is a feed-forward network that learns the context of a word by scanning a sentence in both directions. Thus, BERT would produce a different embedding (or numerical representation) for similarly sounding sentences, like ‘She is watching TV’ and ‘She is watching her kids grow’.

In the context of cybersecurity logs, such embedding can be helpful to distinguish between IP addresses, network endpoints, ports, or free flow comments or messages. In fact, by using BERT embeddings one can train a model to detect such entities. Enter cyBERT!

cyBERT is an automatic tool to parse logs and extract relevant information. To get started, we just need to load the model we intend to use:

cybert = Cybert()
cybert.load_model(
      'pytorch_model.bin'
    , 'config.json'
)

The pytorch_model.bin is a PyTorch model that was trained to recognize entities from Apache WebServer logs; it can be downloaded from the models.huggingface.co/bert/raykallen/cybert_apache_parser S3 bucket. In the same bucket we can find the config.json file.

Once we have the model loaded, CyBERT will utilize the power of NVIDIA GPUs to parse the logs at rapid speed to extract the useful information and produce a structured representation of the log information. The API makes it really simple.

logs_df = cudf.read_csv(‘apache_log.csv')
parsed_df, confidence_df = cybert.inference(logs_df["raw"])

The first dataframe returned contains all the parsed fields from the logs.

while the confidence_df DataFrame outlines how confident the CyBERT model is about each extracted piece of information.

As you can see the model is pretty confident and a glimpse at the data confirms that the extracted information matches the column name.

Want to try other functionality of CLX or simply run through the above examples? Go to the CLX cheatsheet here!

Categories
Misc

NVIDIA Deep Learning Institute Announces Public Workshop Summer Schedule

NVIDIA DLI Summer WorkshopsWorkshops are conducted live in a virtual classroom environment with expert guidance from NVIDIA-certified instructors.NVIDIA DLI Summer Workshops

Training for Success

Continuing its popular public workshop series, the NVIDIA Deep Learning Institute (DLI) released the schedule for June, July, and August of 2021. These workshops are conducted live in a virtual classroom environment with expert guidance from NVIDIA-certified instructors. Participants have access to fully configured GPU-accelerated servers in the cloud to perform hands-on exercises. 

Learning new and advanced software development skills is vital to staying ahead in a competitive job market. DLI offers a comprehensive learning experience on a wide range of important topics in AI, data science, and accelerated computing. Gain hands-on experience with the most widely used, industry-standard software, tools, and frameworks. Successful completion of the course and assessment earns an NVIDIA certificate of competency.

Among the workshops scheduled are two new courses: Accelerating Data Engineering Pipelines and Building Conversational AI Applications. 

To register, visit our website. Space is limited so we encourage you to sign up early.

Here is our current public workshop schedule:

June

Fundamentals of Accelerated Data Science
Tue, June 22, 9:00 a.m. to 5:00 p.m. CEST (EMEA)
Wed, June 23, 9:00 a.m. to 5:00 p.m. PDT (NALA)

Building Intelligent Recommender Systems
Wed, June 23, 9:00 a.m. to 5:00 p.m. CEST (EMEA)
Thu, June 24, 9:00 a.m. to 5:00 p.m. PDT (NALA)

July

Accelerating Data Engineering Pipelines
Tue, July 6, 9:00 a.m. to 5:00 p.m. CEST (EMEA)
Tue, July 13 9:00 a.m. to 5:00 p.m. PDT (NALA)

Building Transformer-Based Natural Language Processing Applications 
Wed, July 7, 9:00 a.m. to 5:00 p.m. CEST (EMEA)

Fundamentals of Accelerated Computing with CUDA Python
Wed, July 14, 9:00 a.m. to 5:00 p.m. PDT (NALA)

August

Building Conversational AI Applications
Tue, August 24, 9:00 a.m. to 5:00 p.m. PDT (NALA)
Tue, August 31, 9:00 a.m. to 5:00 p.m. CEST (EMEA)

Fundamentals of Deep Learning
Wed, August 25, 9:00 a.m. to 5:00 p.m. PDT (NALA)

Fundamentals of Accelerated Computing with CUDA Python
Thu, August 26, 9:00 a.m. to 5:00 p.m. CEST (EMEA)

Visit the DLI website for details on each course and the full schedule of upcoming instructor-led workshops, which is regularly updated with new training opportunities.

For more information, email nvdli@nvidia.com.

Categories
Misc

Unreal Engine 5 Early Access Available Now with DirectX Raytracing, NVIDIA DLSS, and NVIDIA Reflex Support

Unreal Engine 5 (UE5) is available in Early Access, delivering the next-generation engine from Epic Games that will further propel the industry forward.

NVIDIA RTX, Unreal Engine 5 Define Future of Game Development and Content Creation

Today, Unreal Engine 5 (UE5) is available in Early Access, delivering the next-generation engine from Epic Games that will further propel the industry forward.

Unreal Engine is used by more than 11 million creators, making it one of the most popular game engines in the world, and one that continuously pushes the boundaries of what’s possible with real-time technology.  UE5 represents a generational leap in both workflows and visual fidelity, extending the engine’s support for DirectX Raytracing, NVIDIA DLSS, and NVIDIA Reflex, and adding new features such as Nanite and Lumen that make it faster and easier for games to implement photorealistic visuals, large open worlds and advanced animation and physics.

In 2018, NVIDIA launched our RTX technology alongside a stunning Star Wars demo called Reflections, which was built on Epic Games’ Unreal Engine 4. This laid out a vision of a new era of computer graphics for video games that featured photorealistic, ray-traced lighting, AI-powered effects and complex worlds with massive amounts of geometry and high-resolution textures.

Enabling this vision were RTX GPUs with dedicated cores for ray tracing and AI, as well as new hardware capabilities for increased geometric detail and texture streaming. The RTX games that have been released over the last three years, including Fortnite, Metro Exodus and Cyberpunk 2077, have stepped us closer to this vision.

Dynamic Open Worlds Full of Geometric Detail

With the introduction of Nanite and Lumen in UE5, developers can create games that contain massive amounts of geometric detail with fully dynamic global illumination.

Nanite enables film-quality source art consisting of millions or billions of polygons to be directly imported into Unreal Engine — all while maintaining a real-time frame rate and without sacrificing fidelity. Nanite intelligently streams and processes only the detail you can perceive, largely removing poly count and draw call constraints, and eliminating time-consuming work like baking details to normal maps and manually authoring levels of detail. This allows users to focus less on tedious tasks and more on creativity.

With Lumen, developers can create more dynamic scenes where indirect lighting adapts on the fly, such as changing the sun angle with the time of day, turning on a flashlight or opening an exterior door. Lumen removes the need for authoring lightmap UVs, waiting for lightmaps to bake or placing reflection captures, which results in crucial time savings in the development process.

UE5 is making it easier to develop expansive open worlds and provides developers with the GPU-accelerated tools to better animate characters and build audio pipelines.

“We’ve utilized RTX GPUs extensively throughout the development of Unreal Engine 5 and all of the respective sample content released today,” said Nick Penwarden, Vice President of Engineering at Epic Games. “Thanks to a tight integration with NVIDIA’s tools and technologies our team is able to more easily optimize and stabilize UE5 for everyone.”

NVIDIA DLSS and Reflex in Unreal Engine 5

NVIDIA is working with Epic Games to enable a broad suite of technologies in UE5, starting with two of our most popular features — Deep Learning Super Sampling (DLSS) and NVIDIA Reflex.

DLSS taps into the power of a deep learning neural network to boost frame rates and generate beautiful, sharp images. Reflex aligns CPU work to complete just in time for the GPU to start processing, minimizing latency and improving system responsiveness.

DLSS source code and NVIDIA Reflex is available now in Unreal Engine 5, and we will be releasing the DLSS plugin for UE5 in the coming weeks.

Learn more about DLSS, and experience it by downloading the NVIDIA RTX Technology Showcase.

Get more information about NVIDIA Reflex and its impact in competitive games by watching this on-demand GTC session.

Take Unreal Engine 5 for a Test Drive on RTX GPUs Today

Developers can download Unreal Engine 5 Early Access from Epic Games here. Grab an NVIDIA RTX GPU and install the latest NVIDIA graphics driver on your system for the best experience.

With the processing power of RTX GPUs and the next-generation Unreal Engine 5, there’s no limit to what you can create.

Categories
Misc

Does Volatility Harvesting Really Work?

Categories
Offsites

Does Volatility Harvesting Really Work?

Categories
Misc

Installing TensorFlow on Windows with Cuda

I’ve been trying to start the mit deep learning lab1. I know it comes with the notebook you can run in google colab, but I’d like to be able to play with this locally and hopefully experiment easier. I had a ton of problems getting this to work, but I finally did it. Here are some notes.

  1. Install python 3.8 from the python website not from windows app marketplace. I was finding that the windows app one seems to be sandboxed and cannot fully find stuff in PATH.
  2. I grabbed tensorflow nightly as I could not get the default to work at this time.
  3. I used cuda 11.0 update 2. (This is the one I have working other cuda 11s might work). This was ” cuda_11.0.3_451.82_win10.exe ”
  4. I used cuDNN ” cudnn-11.0-windows-x64-v8.0.4.30.zip ” use ” cudnn-11.2-windows-x64-v8.1.0.77.zip ” now. Had a version error later during running a NN but not during install.
  5. nVidia cuDNN check that path is to do the bin folder. Settings -> System -> About -> Advanced System Settings. Advanced tab -> Environment variables.To compare my PATH looks like:C:toolscudabin (this should be the only one you need to manually add)C:Program FilesNVIDIA GPU Computing ToolkitCUDAv11.0binC:Program FilesNVIDIA GPU Computing ToolkitCUDAv11.0extrasCUPTIlib64C:Program FilesNVIDIA GPU Computing ToolkitCUDAv11.0includeC:Program FilesNVIDIA GPU Computing ToolkitCUDAv11.0libnvvpC:Program FilesNVIDIA CorporationNsight Compute 2020.1.2
  6. If you do all of this it still wont work, but pay attention to the errors! there is a dll in C:Program FilesNVIDIA GPU Computing ToolkitCUDAv11.0bin that it cannot find.
  7. This dll is infact xyz_10.dll in the folder but python wants to grab xyz_11.dll. I renamed this to _11.dll and all works. Note xyz is some random name as I for get the exact file name now, but there was only one.

Also I am running this on a non AVX supported cpu (X58 era W3690) with at GTX 1060 6GB. Hope this helps I found a lot of similar posts in various sites with abandonment or no answers and almost gave up. Reposted this as I forgot to join before submitting and this was marked as spam.

submitted by /u/robotStefan
[visit reddit] [comments]

Categories
Misc

Having trouble using customized generator in fit validation

I’m writing a customized generator DataGenerator for model.fit(). It works good if I don’t specify validation_data parameter of model.fit().

python model.fit(training_generator.data_generator(), steps_per_epoch=training_generator.steps, epochs=EPOCHS, verbose=VERBOSE)

If I try to specify validation_data parameter like following

python model.fit(training_generator.data_generator(), steps_per_epoch=training_generator.steps, validation_data=validation_generator.data_generator(), validation_steps=validation_generator.steps, epochs=EPOCHS, verbose=VERBOSE)

I will get the warning W tensorflow/core/kernels/data/ge nerator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled.

Train for 375 steps, validate for 93 steps Epoch 1/200 375/375 [==============================] – 1s 3ms/step – loss: 1.3730 – accuracy: 0.6803 – val_loss: 0.8896 – val_accuracy: 0.8324 Epoch 2/200 372/375 [============================>.] – ETA: 0s – loss: 0.7859 – accuracy: 0.83382021-05-25 13:37:10.626497: W tensorflow/core/kernels/data/ge nerator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled 375/375 [==============================] – 1s 2ms/step – loss: 0.7854 – accuracy: 0.8338 – val_loss: 0.6558 – val_accuracy: 0.8585 Epoch 3/200 373/375 [============================>.] – ETA: 0s – loss: 0.6376 – accuracy: 0.85292021-05-25 13:37:11.257041: W tensorflow/core/kernels/data/ge nerator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled 375/375 [==============================] – 1s 2ms/step – loss: 0.6371 – accuracy: 0.8530 – val_loss: 0.5558 – val_accuracy: 0.8727

Is there any wrong with my generator?

Full code is

“`python import numpy as np import tensorflow as tf from tensorflow import keras

Network and training parameters

EPOCHS = 200 BATCH_SIZE = 128 VERBOSE = 1 CLASSES_NUM = 10 # Number of outputs = number of digits VALIDATION_SPLIT=0.2 # How much TRAIN is reserved for VALIDATION

class DataGenerator: def init(self, x, y, classes_num, batch_size=32, shuffle=True): ‘Initialization’ self.x = x self.y = y 

 self.batch_size = batch_size self.classes_num = classes_num self.shuffle = shuffle self.steps = int(np.floor(len(self.x) / self.batch_size)) def __iter__(self): indexes = np.arange(len(self.x)) if self.shuffle == True: np.random.shuffle(indexes) for start in range(0, len(self.x), self.batch_size): end = min(start + self.batch_size, len(self.x)) idxes = indexes[start:end] batch_x = [self.x[idx] for idx in idxes] batch_y = [self.y[idx] for idx in idxes] yield np.array(batch_x), tf.keras.utils.to_categorical(batch_y, num_classes=self.classes_num) def data_generator(self): while True: yield from self.__iter__()

Loading MNIST dataset

mnist = keras.datasets.mnist (x_train, y_train), (x_test, y_test) = mnist.load_data()

You can verify that the split between train and test is 60,000, and 10,000 respectively.

assert x_train.shape == (60000, 28, 28) assert x_test.shape == (10000, 28, 28) assert y_train.shape == (60000,) assert y_test.shape == (10000,)

X_train is 60000 rows of 28×28 values –> reshaped in 60000 x 784

RESHAPED = 784

x_train = x_train.reshape(x_train.shape[0], RESHAPED) x_test = x_test.reshape(x_test.shape[0], RESHAPED) x_train = x_train.astype(‘float32’) x_test = x_test.astype(‘float32’)

Normalize inputs to be within in [0, 1].

x_train /= 255 x_test /= 255

total = len(x_train)//BATCH_SIZE pivot = int((1-VALIDATION_SPLIT)*len(y_train)) x_train, x_validation = x_train[:pivot], x_train[pivot:] y_train, y_validation = y_train[:pivot], y_train[pivot:]

training_generator = DataGenerator(x_train, y_train, CLASSES_NUM, BATCH_SIZE) validation_generator = DataGenerator(x_validation, y_validation, CLASSES_NUM, BATCH_SIZE)

Build the model

model = tf.keras.models.Sequential() model.add(keras.layers.Dense(CLASSES_NUM, input_shape=(RESHAPED,), name=’dense_layer’, activation=’softmax’))

Summary of the model

model.summary()

Compile the model

model.compile(optimizer=’SGD’, loss=’categorical_crossentropy’, metrics=[‘accuracy’])

model.fit(training_generator.data_generator(), steps_per_epoch=training_generator.steps, validation_data=validation_generator.data_generator(), validation_steps=validation_generator.steps, epochs=EPOCHS, verbose=VERBOSE)

y_test = tf.keras.utils.to_categorical(y_test, CLASSES_NUM)

Evaluate the model

test_loss, test_acc = model.evaluate(x_test, y_test) print(‘Test loss:’, test_loss) print(‘Test accuracy:’, test_acc)

Make prediction

predictions = model.predict(x_test) “`

submitted by /u/Ynjxsjmh
[visit reddit] [comments]