WASHINGTON—One of the world’s largest cryptocurrency exchanges has agreed to pay $100 million to resolve a regulatory lawsuit over its failure to follow U.S. rules while allowing Americans to access its trading platform.

BitMEX, which offers leveraged trading in bitcoin and other cryptocurrency derivatives, had been sued by the Commodity Futures Trading Commission last year. Three of the company’s co-founders were named as defendants in that lawsuit, while they were also separately indicted on a charge of failing to use an effective anti-money-laundering program. They have pleaded not guilty to criminal charges, and Tuesday’s civil settlement doesn’t resolve their cases.

BitMEX, which was incorporated in the Seychelles, neither admitted nor denied the allegations but has also agreed to prevent U.S. residents from using its trading services, the CFTC said in a press release Tuesday. BitMEX was one of several overseas exchanges, many of them based in Asia, that became popular with traders globally who wanted to bet on cryptocurrency derivatives.

China’s recent warning on cryptocurrency sent the market in a tailspin. WSJ’s Aaron Back explains why the recent shake-ups in the value of bitcoin, dogecoin, ether and other cryptocurrencies may point to obstacles in mainstream acceptance. Photo: Dado Ruvic/Reuters The Wall Street Journal Interactive Edition

But from 2014 through 2020, BitMEX collected only an email address from users and didn’t verify traders’ identities, according to the Financial Crimes Enforcement Network, a bureau of the Treasury Department. That failure exposed the exchange to risks such as dealing with money launderers, terrorist financiers, and ransomware attackers, according to FinCEN.

BitMEX conducted at least $209 million of transactions with known darknet markets, according to FinCEN, which typically facilitate dealing in illegal drugs, computer-hacking software and counterfeit goods.

Crypto exchanges have increasingly collided with American regulators after years of trying to dodge compliance with many U.S. rules. The Securities and Exchange Commission earlier this week settled a $10 million enforcement action with Poloniex LLC, whose employees had said they wanted to be aggressive about testing the limits of what could be traded without regulatory compliance.

Exchanges that offer certain types of derivatives to U.S. investors must be registered with the CFTC, which enforces rules designed to ensure financial responsibility and limit risk. The CFTC said that BitMEX operated U.S. offices and offered American customers the ability to trade unregulated swaps, a type of contract that must be registered with the federal regulator. The swaps included contracts on bitcoin, ether and litecoin, the CFTC said.

BitMEX made its trading available to both U.S. retail and institutional customers through its website, the CFTC said. The exchange operator was aware that American customers used virtual private networks to get around barriers that were set up to screen out U.S. traders, according to a consent order filed Tuesday.

“This case reinforces the expectation that the digital assets industry, as it continues to touch a broader pool of market participants, takes seriously its responsibilities in the regulated financial industry and its duties to develop and adhere to a culture of compliance,” CFTC Acting Chairman Rostin Behnam said.

The $100 million fine includes $50 million that is to be paid to FinCEN, which seeks to combat money laundering and other financial crimes.

Since the CFTC filed its lawsuit, BitMEX has developed an anti-money-laundering and user-verification program, according to the consent order. A representative for BitMEX said the firm was pleased to reach the settlement and is “committed to becoming a regulated exchange” and “looking to set the benchmarks in this new era for crypto.”

BitMEX’s founders include Arthur Hayes, Benjamin Delo, Samuel Reed and Gregory Dwyer, who were criminally charged last year with one count of violating the Bank Secrecy Act and one count of conspiracy. The CFTC sued Messrs. Hayes, Delo and Reed.

BitMEX CEO Arthur Hayes, speaking at an event in 2017, once worked as a derivatives trader for Wall Street banks.

Photo: Michael Nagle/Bloomberg News

A spokesman for those three men said Tuesday, addressing the CFTC’s claims, that they sought to comply with “law as it developed over time. The actions against Arthur, Ben, and Sam by the U.S. authorities are unfounded and represent an unwarranted overreach. The co-founders look forward to defending themselves in court.”

Mr. Hayes lives in Singapore, according to court records, and has been an ebullient personality on Twitter, celebrating the volatility of crypto assets and feuding with bitcoin skeptics such as economist Nouriel Roubini. He pleaded not guilty to the criminal charges in Manhattan federal court in April and was allowed to remain in Singapore after posting a $1.5 million cash bail, according to court records.

He graduated from the University of Pennsylvania’s Wharton School in 2008, according to an interview he gave to the Milken Institute in 2019. He worked as a derivatives trader for Wall Street banks in Asia for several years and was also an amateur bodybuilder, he wrote in an essay earlier this year.

Write to Dave Michaels at dave.michaels@wsj.com