An image posted on a militant website shows Islamic State fighters marching in Raqqa, Syria. Associated Press

The gruesome James Foley execution video released this week by Islamic State marks the highest-profile offensive by the militant group in a social-media battle with the West that has escalated in recent months.

The Islamic State—also know as the Islamic State of Iraq and al-Sham, or ISIS—has developed a sophisticated social-media strategy, complete with high-resolution videos and hashtag campaigns, striking for their contrast with the grainy Osama bin Laden videos that the al Qaeda leader would periodically release to television networks during his years in hiding.

“It’s an old game that’s being played in a new way,” said P.J. Crowley, who served as assistant secretary of state for public affairs during President Barack Obama‘s first term. He says the Islamic State’s methods, while sophisticated, may backfire due to their gruesomeness.

The execution video released this week by ISIS served to show that the group was serious about its ransom demands and to sow more fear as it fights to establish a caliphate in the Middle East. The video also sparked an intense scramble by social-media services such as YouTube and Twitter TWTR +1.93% to take down images and video of the killing and to slow its spread.

Phillip Smyth, a researcher of jihadist groups at the University of Maryland, said he started seeing a rise of social-media activity by the Islamic State and its sympathizers in June.

The Islamic State’s media strategy shows various sides of the militants. In July, ISIS media arm Al Hayat released a video with English subtitles on social media showing young children breaking their Ramadan fast with militants. A week later, Al Hayat published a video to coincide with the Eid feast closing Ramadan on July 28 that ended with a mass execution. Al Hayat also publishes an online newsletter translated into English.

J.M. Berger, editor of IntelWire.com, has followed jihadists’ social-media efforts for years, and says there are thousands of accounts on Twitter tied to the Islamic State. Some users claim to be militants, others claim to be supporters, and others say they aren’t connected to the group—which also helps them avoid being banned on social media. The accounts repost one another’s tweets and send messages to various government accounts, media or even celebrities with images of beheadings and threats to the U.S.

ISIS backers also take a lighter touch. When Robin Williams died, an Islamist Twitter user and ISIS backer known as @Mujahid4life posted a tweet about loving the movie “Jumanji” as a child. Soon, the jihadist was tweeting about his taste in other movies—”The Lion King” was his favorite Disney DIS +0.12% film. His account has been deleted.

Last week, ISIS supporters started a meme, posing with jars of Nutella in ISIS-occupied areas in Syria and Iraq. Mr. Smyth says these memes are organized and planned in online forums and then rolled out on social networks as a deliberate strategy to make the Islamic State seem more friendly and familiar to Westerners.

The two-faced push makes for shocking contrasts. At about the same time ISIS-tied accounts were tweeting about Nutella, a photo of a child holding a decapitated head was distributed on Twitter, primarily by ISIS supporters. It was quickly condemned by Secretary of State John Kerry, who called the image “utterly disgraceful,” saying “it underscores the degree to which [ISIS] is so far beyond the pale with respect to any standard by which we judge even terrorist groups.”

The State Department engages with terrorists and would-be recruits daily through its Center for Strategic Counterterrorism Communications, which was created by an executive order in 2011. The U.S. initially used online forums to counter ISIS propaganda, but it moved its message to Twitter late last year, sending its first tweet in December 2013. In July, the State Department said it increased its messages targeting the Islamic State. Earlier this week, the Center for Strategic Counterterrorism Communications created a Facebook page.

The efforts target “fence sitters” rather than committed militants, and hope to prevent them from being recruited by the Islamic State and other terrorist organizations, said Will McCants, a former State Department official who helped to create the Center for Strategic Counterterrorism Communications.

“One of the challenges that the State Department faces is how do we get enough eyeballs on what we are saying without drawing attention to people that shouldn’t be drawn attention to,” Mr. McCants said. “You don’t want to engage with every Joe Jihadi out there, because you don’t want to elevate their profile.”

In its Twitter account, Think Again Turn Away, the State Department regularly uses hashtags that the Islamic State has adopted to reach more people, such as #CalamityWillBefallUs. ISIS supporters use the hashtag to tweet pictures of the terror attacks on 9/11. The State Department responds with news articles about U.S. airstrikes.

On Wednesday, when Twitter accounts tweeted that members of the Yazidis religious minority were converting to Islam, Think Again Turn Away tweeted: “Anything to avoid beheading.”

Rank-and-file American social-media users troll ISIS as well. When Islamic State supporters tweeted using #AMessagefromISIStoUS with a picture of an ISIS flag on a cellphone in front of the White House, many Americans used the same hashtag to threaten back, bragging of U.S. military power. When grisly images of Mr. Foley’s beheading spread online, the hashtag #ISISMediaBlackout trended, with users telling people to stop giving ISIS attention and share photos of Mr. Foley’s life instead.

A big challenge for groups like ISIS is keeping content online. Services including Facebook specifically ban terrorist activity in their terms of service and, along with Twitter and YouTube, will work to remove graphic content. On Thursday, State Department spokeswoman Marnie Huff said the State Department itself reported the beheading video to social networks citing its graphic content, not its ISIS connection.

Militants try to distribute content broadly to stay ahead of deletions. By the time one account is deleted, the video and images have already been reposted, creating a game of Whac-A-Mole for the social networks. Militants also shift bases: IntelWire’s Mr. Berger said official ISIS accounts have been deleted from Twitter and he has seen roughly 90 ISIS-related accounts disappear. But he’s now seeing ISIS pop up on the open-source network Friendica and on diaspora*, a software-based decentralized network.

On Wednesday, diaspora* posted a blog post saying it has removed accounts connected to ISIS and will continue to monitor the group.