Featuring developments in Federal highway policies, programs, and research and technology. 
This magazine is an archived publication and may contain dated technical, contact, and link information. 
Federal Highway Administration > Publications > Public Roads > Vol. 72 · No. 2 > Using Supercomputers to Determine Bridge Loads 
Sept/Oct 2008 

Publication Number: FHWAHRT08006 

Using Supercomputers to Determine Bridge Loadsby Kornel Kerenyi, Tanju Sofu, and Junke GuoFHWA and the Argonne National Laboratory are using multidimensional programs to study hydrodynamic forces on flooded bridge decks.
Bridges are a vital component of the Nation's transportation network. Evaluating their stability and structural response after flooding is critical to highway safety. When a bridge crossing a waterway is partially or entirely submerged during a flood, the water can exert significant loading on the deck and threaten its integrity. Being able to estimate hydrodynamic loading accurately can help designers and transportation agencies build better, stronger bridges. Accurately modeling turbulence and sediment transport is essential for estimating the impact of scour on bridges. Researchers typically address problems in science and engineering through two complementary approaches: experimental and analytical (or theoretical). In many applications, such as the fluid mechanics of streams and the impacts on bridges, the governing equations are nonlinear and, except in special circumstances, analytical solutions are not available. In addition, fluid mechanics applications often are multidimensional in nature and timedependent, which further complicates attempts to understand and model reallife turbulence and scour conditions. Pursuing a third approach, researchers at the Federal Highway Administration (FHWA) and the new Transportation Research and Analysis Computing Center (TRACC) in West Chicago, IL, a partnership of the U.S. Department of Transportation (USDOT) and the Argonne National Laboratory, are using supercomputers with multidimensional hydraulics programs to closely mimic reallife conditions. The researchers verify the computers' output using flume models at FHWA's TurnerFairbank Highway Research Center (TFHRC) in McLean, VA, indicating that computers might play an even more valuable role in bridge hydraulics and safety. "As computer technology improves and the ability to accurately model turbulence and sediment transport becomes more robust, transportation agencies will benefit in both the planning and design of bridges through improved hydraulic and scour estimation," says Kevin Flora, hydraulics branch chief for structure maintenance and investigation at the California Department of Transportation (Caltrans). The Third ApproachEffective use of prototypic experiments is a key approach to understanding reallife phenomena. For many fluid dynamics applications, such as those associated with bridge hydraulics, fullscale tests are not possible. Researchers use smaller scale, and perhaps simplified, representations of the physical configuration, and they extrapolate the results to apply to actual conditions. Some uncertainty remains in this extrapolation, however, related to the use of simplified experiments to predict the behavior of complex physical systems. Computational fluid dynamics (CFD) attempts to address these issues and complement the experimental and analytical approaches through numerical solutions. CFD is a branch of fluid mechanics that uses numerical methods and algorithms to analyze and solve problems involving fluid flows, such as water. Researchers use computers to perform the millions of calculations necessary to simulate the interaction of fluids with the complex surfaces involved in engineering. CFD enables scientists and engineers to perform numerical simulations in the laboratory and significantly reduce the amount of experimentation and overall cost. CFD is a highly interdisciplinary research area at the interface of physics, applied mathematics, and computer science. CFD calculations are numerical solutions to the underlying equations that represent the flow of fluids. As a result, a computational grid — focusing many computers in a network on a single problem — can approximate the true geometry of a given physical condition. In the CFD approach, researchers can include actual observed boundary conditions that might be impossible to represent in a laboratory experiment and perform parametric studies on material properties and physical conditions that might be expensive or timeconsuming to perform experimentally. Although CFD can complement laboratory experiments and theoretical approaches, generating accurate numerical solutions requires establishing finemesh computational grids to represent the actual geometry of the problem. Also, in timedependent cases, the accuracy and numerical stability of the solutions often require small time steps to capture the influence of temporal variations in the flow field. Computing solutions on these grids can require many calculations, so the use of computers is essential. Development of powerful supercomputers, including massively parallel computers (which contain hundreds of processors sharing a common hardware platform), and adaptation of the CFD techniques for use with these supercomputers in recent years is enabling engineers and scientists to perform these complex CFD calculations. "By comparing the computed results to some experimental information or to analytical solutions under simplified conditions, analysts can verify and validate the numerical approach," explains TRACC Director David P. Weber. "With verified and validated results, analysts can then feel much more confident in performing calculations on more complex representations of the true physical conditions. Thus, the three approaches of experiment, analysis, and computation are complementary." Establishing TRACCThe Argonne National Laboratory is leading the initiative to establish a highperformance computing center, TRACC, to pursue transportation research and development (R&D) programs. Argonne analysts provide technical support to researchers at TFHRC on CFD simulations.
The overall objective of this effort is to establish validated computational practices to address the transportation community's research needs in bridge hydraulics. Traditionally, the bridge hydraulics work relies on scaled experiments to provide measurements for flow field, which is the velocity and turbulence of a fluid as functions of position and time. Now, however, parallel computers and commercially available software provide an opportunity to shift the focus of these evaluations to the CFD domain. After being validated using the data from a limited set of experiments, highfidelity CFD simulations can be used to expand the range and scope of parametric studies. The CFD simulations also can be used to predict the effects of scaling by studying differences between the reducedscale experiments and fullscale bridges. Most recently, reducedscale experiments conducted at TFHRC's J. Sterling Jones Hydraulics Research Laboratory established the foundations of a CFDbased simulation methodology. Researchers at Argonne and TFHRC worked together to study CFD techniques for simulating openchannel flow around inundated bridges. "The use of supercomputers and CFD code can potentially overcome the inherent problems of modeling sediment in the laboratory," says Flora. "Once physically based models of sediment transport and scour are developed, computer simulations will provide engineers with new insights into better design and mitigate for scour at structures." Research ConsiderationsThe ArgonneTFHRC research included analysis of lift forces (FL) (or simply lift), forces produced perpendicular to the flow of a fluid; drag forces (FD), forces exerted on objects in the path of fluids; and moments (M), the tendency to cause rotation about a point or axis, on inundated bridge decks under various flow configurations. The two research teams investigated the applicability of commercial CFD software for predicting flow field and evaluating lift, drag forces, and moments. They compared the results with laboratory data using an ultraprecise force balance (which is a mounting device to measure hydrodynamic forces on scale model bridge decks) to measure lift, drag force, and moments for various inundation ratios and flow rates. They also used Particle Image Velocimetry (PIV) to study flow fields around submerged model bridge decks. PIV is a noninvasive measurement technique to visualize flow distributions. The technique involves adding microscopically small, highly reflective particles to the flow and using a laser to illuminate a thin layer of the flow, so only the particles in that light sheet reflect the laser's light. Using cameras pointed with different angles toward the light sheet, researchers can capture images and an algorithm based on statistical probability to determine the speed and direction of the moving particles. PIVrelated tasks included the following:
The researchers also adopted numerical simulations using two turbulence models, highReynolds number k(epsilon) model and large eddy simulation (LES), to resolve unsteady turbulent flow. Turbulent flow is an irregular condition in which the flow particles show a random variation with time and space coordinates. The k(epsilon) model simulates transport of both the turbulent kinetic energy (k) and the turbulent energy dissipation rate (epsilon). The rationale for using LES is to simulate the larger and more easily resolvable scales of the motions while accepting that LES will not represent the smaller scales accurately. The researchers examined the agreement between experimental data and the results of CFD simulations for a typical U.S. highway girder deck, corresponding to Froude numbers (dimensionless ratio) ranging from 0.12 to 0.40 characterizing the openchannel flow. That is, the Froude number is a ratio comparing inertial and gravitational forces. If the Froude number is less than 1, the flow is subcritical, and if it is greater than 1, the flow is supercritical. The TRACC researchers used STARCD (software for modeling fluid dynamics), while the TFHRC team used Fluent, Inc. CFD software in their simulations. The researchers integrated forces and moments over the surface of the bridge deck along the flow and perpendicular directions, respectively. When calculating the lift over the bridge deck, however, they excluded the component of buoyancy, as they were interested only in hydrodynamic loading, not static loading. Advances in CFD research provide the basis for modeling the multiphase nature of openchannel flow that consists of air and water separated by a free surface. In multiphase flows, each phase has individually defined physical properties and flow fields. In both the STARCD and Fluent approaches, the researchers had three multiphase flow models available. The first was the volume of fluid (VOF) model, which can model two or more nonmixable fluids by solving a single set of continuum equations and tracking the volume fraction of each of the component fluids throughout the domain. Applications of the VOF model include stratified flows, freesurface flows, filling, sloshing, the motion of large bubbles in a liquid, the motion of liquid after a dam break, the prediction of jet breakup (surface tension), and the steady or transient tracking of any liquidgas interface. In the VOF model, in addition to the conservation equation for mass and momentum, one has to introduce the volume fraction of phase in computational cell. In each control volume, the volume fraction of all phases sum to unity. The fields for all flow variables are shared by the phases, as they represent volumeaveraged values. Thus, the variables and properties in any given cell are representative of one of the phases, or a mixture of phases, depending upon the volume fraction values. Second was the mixture model, in which the flow consists of a continuous phase and one or more dispersed phases. Although it is computationally efficient, this Lagrangian approach is suitable only for dispersed flows and cannot be used to accurately resolve the stratified nature of freesurface flows. The third was the Eulerian model, which is a multiphase model in which the phases are treated as interpenetrating continua coexisting in the flow domain. The Eulerian model provides a general framework for both dispersed and stratified multiphase flows. However, the free surface calculated with this approach is usually less sharp in comparison with the VOF method, making the latter the preferred model. Therefore, the researchers are testing the VOF computational method with both STARCD and Fluent to capture the impact of free surface reasonably accurately and efficiently.
Experimental Setup At TFHRCThe TFHRC researchers conducted the laboratory experiments using a 12.8meter (42foot)long, 0.4meter (1.3foot)wide, and 0.5meter (1.6foot)high Plexiglas® rectangular flume. They set the flume horizontally and controlled the depth of flow with an automatic adjustable tailgate located at the downstream end. The team used a 0.054cubicmeter (2.0cubicfeet)persecond pumping system to supply the flow. They measured the water surface level using ultrasonic sensors at two cross sections along the flume. An electromagnetic flow meter measured the discharge. They used an Acoustic Doppler Velocimeter probe to measure the velocity distribution (speed of the water) of the flow. The researchers constructed a sixgirder bridge deck shape, which is typical of U.S. highways, using yellow Plexiglas and adopting a geometrical reduction scale of 1:40 based on the depths, maximum discharges (flow rate), and inundations (submergence level) available. This scaled bridge deck is optimal to produce values ranging from low to high flows, all at subcritical Froude numbers (less than 1) in the upstream flow. To measure forces induced by the bridge deck model in the flow direction (drag) and simultaneously perpendicular to the flow direction (lift) using electric strain gauges, the research team constructed an ultraprecise force balance, which can capture even small forces. They mounted the bridge deck model between brackets attached to a measuring platform. To test different submergence ratios, they mounted the platform and model flexibly to allow vertical positioning of the bridge deck. The flexibility was important to perform testing at various submergence levels of the bridge deck. They conducted the experiments for water approach velocities ranging from 0.25 to 0.50 meter (0.8 foot to 1.6 feet) per second. They kept the flow depth for all experiments constant at 0.25 meter (0.8 foot). They varied the Froude number within a range of 0.16 to 0.32 and the submergence of the bridge deck from slight submergence of the girders to complete overtopping of the bridge deck, inundation ratio, h* = 0.29 to 3.2. The force balance experiments used the PIV technique to visualize and measure flow fields for the submerged bridge deck model. The PIV technique is an optical flow diagnostic based on the interaction of light refraction and scattering, using nonhomogeneous media. The fluid motion is made visible by tracking the locations of small tracer particles at two instances of time. The researchers then use the particle displacement as a function of time to infer the velocity flow field. They analyzed the data in a format that could be compared with the CFD modeling. TRACC Computing FacilityTRACC is a general purpose advanced computing and visualization facility available to the transportation community for a broad spectrum of applications. Discussions among staff at USDOT's Research and Innovative Technology Administration and FHWA identified specific initial applications and technologies for assigning the highest priority for research and development (R&D) and user support. The TRACC components include highperformance computing, visualization, and networking systems. To take advantage of Argonne's extensive experience in acquiring and operating similar facilities, TRACC acquired the system components and then set them up in dedicated facilities at the DuPage National Technology Park near the DuPage Airport in Illinois. The TRACC computational cluster is a 512core, customized LS1 system from Linux Networx that comprises 128 computational nodes, each with two dualcore Advanced Micro Devices, Inc. OpteronTM 2216 central processing units and 4 gigabytes of random access memory; a DataDirectTM Networks storage system consisting of 240 terabytes of shared RAID (Redundant Array of Inexpensive (or Independent) Disks or (Drives)) storage that is expandable to 750 terabytes; a highbandwidth, lowlatency InfiniBand network for internode computations; and a highbandwidth Gigabit Ethernet management network. TRACC also provides scientific visualization capabilities through the National Center for Supercomputing Applications' Technology, Research, Education, and Commercialization Center at the same location. TRACC meets the needs for visualization of multidimensional data via a highperformance graphics cluster linked with a 15panel liquid crystal display (LCD) tiled display and a portal optimized for visual simulation and highspeed broadband connectivity. Test ResultsThe researchers conducted the CFD simulations using models of an 8.0meter (26.2foot)long, 0.34meter (1.1foot)wide, 0.5meter (1.6foot)high rectangular channel. The study simulated both 2D and 3D models. The researchers placed the bridge deck 4.4 meters (14.4 feet) downstream of the inlet of the channel to obtain a fully developed flow upstream. They situated the outlet far away from the deck to avoid the influence of the outlet boundary on the flow field around it. The team conducted the simulations using Fluent and STARCD for approach velocities ranging from 0.25 meter (0.8 foot) to 0.50 meter (1.6 feet) per second. The Froude number varied within the range of 0.16 to 0.32, and the team ran the simulations for various inundation ratios. The researchers modeled the flow domain using either tetrahedral cells (in Fluent) or hexahedral cells (in STARCD), both with gradually refined mesh structure in the vicinity of the bridge deck. (The gradual refinement of mesh in the vicinity of the deck reduced computational time. If the researchers had refined the mesh in the complete domain, it would have increased the number of cells, which would require more computational time.)
The team presented the experimental and simulation results in dimensionless form for various Froude numbers (which characterize the openchannel flow) and the inundation ratio (which characterizes the flooding height). They presented the measured and simulated forces and moments in terms of dimensionless coefficients as drag, lift, and moment coefficients. They performed the Fluent simulations for 200 seconds using LES as well as the k(epsilon) turbulence model. Due to fluctuating values of CD (drag force coefficient), CL (lift force coefficient), and CM (moment coefficient) with time, they averaged the final values from 50 to 200 seconds. They checked the velocity distribution along the depth of flow at various locations upstream of the bridge deck, and profiles clearly showed that fully developed flow occurs upstream of the bridge deck.
The experimental data can be compared with the Fluent and STARCD simulation results for the drag coefficient as a function of dimensionless inundation ratio, h*, and Froude number, Fr, which characterizes the flow rate. The simulated drag coefficient using Fluent increases as the inundation ratio increases from 1.0 to 1.5, and then tends to level off for the inundated bridge deck. In other words, increasing the water level over the submerged bridge causes CD to reach its maximum and then stay constant, similar to what the researchers observed in the experiments. Although the simulated data using STARCD are slightly above those observed from Fluent, they are generally compatible with each other. In practice, an empirical equation can be used to estimate CD, including the effects of the inundation ratio and Froude number.
The researchers also looked at the variation of lift coefficient values, CL, versus inundation ratio using Fluent and STARCD simulation results along with the experimental data. Like the experiments, the component of buoyancy is excluded from the lift forces calculated with the CFD models. Both the simulated and experimental lift coefficients come out to be negative, indicating that the net hydrodynamic force is directed downward. The lift coefficient value decreases as the inundation ratio decreases (that is, it gets more negative) until h* = 1.0. The maximum (negative) value of CL is observed between h* = 0.5 and 1.0. Like the drag coefficient, when inundation is very large, the effect of free surface diminishes and CL tends to a constant. This means that when the bridge deck is immersed well below the water level, the net lift force on the bridge deck does not change. The researchers also compared the moment results of the drag and lift forces for the simulation data and experimental data. Again, the simulated results from Fluent and STARCD were close to the experimental results. The research team conducted the PIV experiments in a special flume using transparent Plexiglas models. The PIV flow field analysis was necessary to calibrate the CFD models. The researchers observed strong agreement between computer simulation and PIV experiments; comparison of the contracted flow fields under the bridge deck especially showed excellent conformity. Significant FindingsThe k(epsilon) turbulence model and LES together with the VOF method simulates faithfully the flow past the bridge deck in an open channel. The predictions of drag, lift, and moment coefficients through the numerical modeling show a trend similar to the flume experimental results. Future bridge designers can use the numerical CFD model without much effort to obtain the coefficients on a bridge deck in an open channel for various flow conditions encountered in practice. The Argonne and FHWA researchers are undertaking additional simulations for other shapes of bridge decks to predict the drag, lift, and moments with the help of CFD. "The validation of the computer modeling gives confidence in the results from computer program analysis of complex hydraulics structures without using physical models," says Michael Fazio, deputy director of research and innovation at the Utah Department of Transportation. "This is a great advancement for hydraulics research." FHWA's future strategic plan for hydraulics R&D proposes to move away gradually from physical experiments and use more CFD modeling to develop design guidance for practitioners. This successful collaboration between Argonne and FHWA's hydraulics R&D program is the first step toward that vision. "The use of CFD software on highspeed computers will undoubtedly play an increasingly important role in the future for understanding scour potential at bridges," says Caltrans's Flora. "Rapid development of various bridge scenarios will be much easier through numeric simulation on supercomputers than through physical modeling in the lab."
Kornel Kerenyi is a hydraulic research program manager in FHWA's Office of Infrastructure R&D. He coordinates FHWA's hydraulic and hydrology research activities with State and local agencies, academia, and various partners and customers, and he manages the Hydraulics Laboratory. He was previously a research engineer for GKY & Associates, Inc. and supervised the support staff in the data collection and analysis for this study. Kerenyi holds a doctorate in fluid mechanics and hydraulic steel structures from the Vienna University of Technology in Austria. Tanju Sofu manages the Engineering Simulation and Safety Analysis section in the Engineering Analysis Department at Argonne. He specializes in largescale computational physics and fluid dynamics simulations on highperformance computing platforms and has extensive experience with a wide range of engineering systems analyses involving multidimensional, multiscale, multiphysics phenomena. Sofu holds a doctorate from The University of Tennessee Knoxville. Junke Guo is an assistant professor in the department of Civil Engineering at the Peter Kiewit Institute at the University of NebraskaOmaha. He received his doctorate in fluid mechanics and hydraulics from Colorado State University. His research interests include CFD, turbulent boundary layer flows, openchannel turbulence and sediment transport, and environmental fluid mechanics. His current research emphasizes CFD applications in transportationrelated flows. For more information, contact Kornel Kerenyi at 2024933142, kornel.kerenyi@dot.gov; Tanju Sofu at 6302629673, tsofu@anl.gov; or Junke Guo at 4025543873, junkeguo@mail.unomaha.edu. 
Page Owner: Office of Corporate Research, Technology, and Innovation Management Scheduled Update: Archive  No Update Technical Issues: TFHRC.WebMaster@dot.gov Updated: 02/20/2015
